2023-12-06T09:01:24.005 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T09:01:24.006 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_sseshasa@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} --archive /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622 --name sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi --verbose -- /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/orig.config.yaml 2023-12-06T09:01:24.052 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T09:01:24.139 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622 branch: wip-sseshasa-testing-2023-12-04-1848 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} email: null first_in_suite: false job_id: '7479622' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-sseshasa-testing-2023-12-04-1848 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - \(SMALLER_PGP_NUM\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW OPS\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sseshasa-testing-2023-12-04-1848 sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 owner: scheduled_sseshasa@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5341 sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 sleep_before_teardown: 0 subset: 222/150000 suite: rados suite_branch: wip-sseshasa-testing-2023-12-04-1848 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHHTCpfjfHqstmkXRAr29MBgjiE9LKJ7mmVdbETF57yMeJIhqBvVHbaKtJ/N+dGuo6+PDjx/O8tQ0aC31Q0dv0= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwTJzvIaMcfYCeVIdChPPodcm3xbZK0f0XHf0RcDpgGKOBNDbbi3jNe6hMMeG4nf8PxhSd4BZBMjCWL9Zi/qqA= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - mon_recovery: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-06_06:16:39 tube: smithi user: sseshasa verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-06T09:01:24.140 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa; will attempt to use it 2023-12-06T09:01:24.140 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks 2023-12-06T09:01:24.140 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-06T09:01:24.142 INFO:teuthology.task.internal:Checking packages... 2023-12-06T09:01:24.165 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '96fa2578c622ca9c169af6f426ca8b1616130e58' 2023-12-06T09:01:24.165 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-06T09:01:24.165 INFO:teuthology.packaging:ref: None 2023-12-06T09:01:24.165 INFO:teuthology.packaging:tag: None 2023-12-06T09:01:24.165 INFO:teuthology.packaging:branch: wip-sseshasa-testing-2023-12-04-1848 2023-12-06T09:01:24.166 INFO:teuthology.packaging:sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T09:01:24.166 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-sseshasa-testing-2023-12-04-1848 2023-12-06T09:01:24.314 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7644.g96fa2578 2023-12-06T09:01:24.314 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-06T09:01:24.321 INFO:teuthology.task.internal:no buildpackages task found 2023-12-06T09:01:24.321 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-06T09:01:24.336 INFO:teuthology.task.internal:Saving configuration 2023-12-06T09:01:24.351 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-06T09:01:24.357 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T09:01:24.383 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 08:55:42.840638', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHHTCpfjfHqstmkXRAr29MBgjiE9LKJ7mmVdbETF57yMeJIhqBvVHbaKtJ/N+dGuo6+PDjx/O8tQ0aC31Q0dv0='} 2023-12-06T09:01:24.404 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 08:55:42.842083', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwTJzvIaMcfYCeVIdChPPodcm3xbZK0f0XHf0RcDpgGKOBNDbbi3jNe6hMMeG4nf8PxhSd4BZBMjCWL9Zi/qqA='} 2023-12-06T09:01:24.404 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-06T09:01:24.411 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-06T09:01:24.411 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-06T09:01:24.411 INFO:teuthology.run_tasks:Running task console_log... 2023-12-06T09:01:24.490 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f898f9f7ee0>, signals=[15]) 2023-12-06T09:01:24.491 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-06T09:01:24.498 INFO:teuthology.task.internal:Opening connections... 2023-12-06T09:01:24.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:01:24.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:01:24.586 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:01:24.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:01:24.676 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-06T09:01:24.687 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-12-06T09:01:24.706 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-12-06T09:01:24.706 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-12-06T09:01:24.762 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2023-12-06T09:01:24.762 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2023-12-06T09:01:24.762 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2023-12-06T09:01:24.762 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-06T09:01:24.763 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T09:01:24.764 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-12-06T09:01:24.790 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2023-12-06T09:01:24.808 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2023-12-06T09:01:24.809 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2023-12-06T09:01:24.866 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T09:01:24.867 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-06T09:01:24.867 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T09:01:24.867 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2023-12-06T09:01:24.890 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-06T09:01:24.901 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-06T09:01:24.908 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-06T09:01:24.908 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-12-06T09:01:24.910 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2023-12-06T09:01:24.932 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-06T09:01:24.938 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-06T09:01:24.938 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-12-06T09:01:24.967 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2023-12-06T09:01:25.051 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-06T09:01:25.110 INFO:teuthology.run_tasks:Running task kernel... 2023-12-06T09:01:25.130 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-06T09:01:25.130 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-06T09:01:25.130 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T09:01:25.131 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T09:01:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:01:25.147 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-12-06T09:01:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:01:25.148 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2023-12-06T09:01:25.203 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-361.el9.x86_64 2023-12-06T09:01:25.203 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-361.el9.x86_64 2023-12-06T09:01:25.203 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2023-12-06T09:01:25.205 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-361.el9.x86_64 2023-12-06T09:01:25.206 INFO:teuthology.task.kernel:Running kernel on smithi132: 5.14.0-361.el9.x86_64 2023-12-06T09:01:25.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2023-12-06T09:01:27.284 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-06T09:01:27.349 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-06T09:01:28.717 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-12-06T09:01:28.731 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2023-12-06T09:01:29.538 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-12-06T09:01:31.916 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-12-06T09:01:33.942 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 19 MB/s | 18 MB 00:00 2023-12-06T09:01:38.067 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 5.8 MB 00:00 2023-12-06T09:01:39.919 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2023-12-06T09:01:40.405 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 18 MB 00:09 2023-12-06T09:01:42.035 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 20 MB 00:01 2023-12-06T09:01:45.424 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 5.8 MB 00:01 2023-12-06T09:01:46.781 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 15 kB 00:00 2023-12-06T09:01:46.943 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 6.1 kB/s | 1.7 kB 00:00 2023-12-06T09:01:47.139 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:01 ago on Wed 06 Dec 2023 09:01:46 AM UTC. 2023-12-06T09:01:48.824 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2023-12-06T09:01:49.191 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-06T09:01:49.191 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-06T09:01:49.306 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:01:49.317 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:01:49.318 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2023-12-06T09:01:49.318 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:01:49.318 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 96 M 2023-12-06T09:01:49.318 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 121 M 2023-12-06T09:01:49.318 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:01:51.386 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 2.9 MB/s | 5.6 MB 00:01 2023-12-06T09:01:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2023-12-06T09:01:55.435 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 3.4 MB/s | 20 MB 00:05 2023-12-06T09:01:55.814 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-06T09:01:55.814 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-06T09:01:55.904 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:01:55.919 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:01:55.919 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T09:01:55.919 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:01:55.919 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:01:55.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-06T09:01:55.920 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:01:55.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-06T09:01:55.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-06T09:01:55.921 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-06T09:01:55.921 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:01:55.921 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:01:55.921 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:01:55.921 INFO:teuthology.orchestra.run.smithi132.stdout:Install 4 Packages 2023-12-06T09:01:55.921 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:01:55.922 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 96 M 2023-12-06T09:01:55.922 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 121 M 2023-12-06T09:01:55.922 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:01:56.077 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 5.7 MB/s | 38 MB 00:06 2023-12-06T09:01:57.025 INFO:teuthology.orchestra.run.smithi132.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 5.3 MB/s | 5.6 MB 00:01 2023-12-06T09:01:57.702 INFO:teuthology.orchestra.run.smithi132.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 12 MB/s | 20 MB 00:01 2023-12-06T09:01:58.443 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 4.6 MB/s | 32 MB 00:07 2023-12-06T09:01:58.443 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:01:58.444 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 96 MB 00:09 2023-12-06T09:01:58.451 INFO:teuthology.orchestra.run.smithi132.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 15 MB/s | 38 MB 00:02 2023-12-06T09:01:58.835 INFO:teuthology.orchestra.run.smithi132.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 18 MB/s | 32 MB 00:01 2023-12-06T09:01:58.835 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:01:58.835 INFO:teuthology.orchestra.run.smithi132.stdout:Total 33 MB/s | 96 MB 00:02 2023-12-06T09:01:59.300 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:01:59.668 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:01:59.668 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:01:59.694 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:02:00.062 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:02:00.062 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:02:00.967 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:02:00.967 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:02:01.444 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:02:01.445 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:02:03.891 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:02:04.096 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:02:04.526 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-06T09:02:04.668 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T09:02:04.729 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-06T09:02:04.869 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T09:02:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T09:02:07.586 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T09:02:07.604 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T09:02:07.766 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T09:02:12.222 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T09:02:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:12.389 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T09:02:12.515 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:16.952 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:17.128 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:45.826 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:45.867 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:45.933 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:45.975 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:46.242 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:46.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-06T09:02:46.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T09:02:46.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T09:02:46.430 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:46.430 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-06T09:02:46.430 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T09:02:46.430 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:02:46.619 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:02:46.832 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T09:02:46.832 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:02:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:02:47.154 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T09:02:47.293 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T09:02:47.294 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.294 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.294 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T09:02:47.294 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T09:02:47.294 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-06T09:02:47.294 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T09:02:47.294 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T09:02:47.332 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T09:02:47.413 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T09:02:47.413 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2023-12-06T09:02:47.446 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 68.el9 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8264254 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-06T09:02:47.447 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:02:47.448 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T09:02:47.449 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T09:02:47.449 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.449 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T09:02:47.483 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T09:02:47.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.483 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.483 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T09:02:47.483 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T09:02:47.483 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-06T09:02:47.483 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T09:02:47.483 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T09:02:47.600 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T09:02:47.601 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2023-12-06T09:02:47.643 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-06T09:02:47.644 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:02:47.645 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T09:02:47.646 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T09:02:47.646 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 5.14.0-390.el9.x86_64 2023-12-06T09:02:47.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T09:02:48.080 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2023-12-06T09:02:48.279 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2023-12-06T09:02:49.492 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-06T09:02:49.531 INFO:teuthology.orchestra.run.smithi088.stderr:done 2023-12-06T09:02:49.539 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-12-06T09:02:49.553 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.PnyOtA7RUV 2023-12-06T09:02:49.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PnyOtA7RUV 2023-12-06T09:02:49.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.PnyOtA7RUV 2023-12-06T09:02:49.662 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-06T09:02:49.682 INFO:teuthology.orchestra.run.smithi132.stderr:done 2023-12-06T09:02:49.684 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2023-12-06T09:02:49.699 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.sHzhRMJQbd 2023-12-06T09:02:49.700 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sHzhRMJQbd 2023-12-06T09:02:49.754 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.PnyOtA7RUV is 6KB 2023-12-06T09:02:49.774 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.sHzhRMJQbd 2023-12-06T09:02:49.803 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.PnyOtA7RUV 2023-12-06T09:02:49.816 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T09:02:49.881 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-06T09:02:49.882 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-06T09:02:49.882 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-06T09:02:49.882 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-06T09:02:49.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-06T09:02:49.913 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.sHzhRMJQbd is 6KB 2023-12-06T09:02:49.962 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.sHzhRMJQbd 2023-12-06T09:02:49.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T09:02:50.031 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2023-12-06T09:02:50.040 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-06T09:02:50.040 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-06T09:02:50.041 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-06T09:02:50.041 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-06T09:02:50.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-06T09:02:50.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2023-12-06T09:03:20.062 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-06T09:03:20.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:03:20.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:03:20.210 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-06T09:03:20.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T09:03:20.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:03:38.594 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-12-06T09:03:38.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-12-06T09:03:47.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:03:47.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:03:47.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T09:03:47.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:03:50.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-12-06T09:04:02.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:04:02.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:04:05.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-12-06T09:04:20.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:04:20.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:04:39.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-12-06T09:04:47.615 DEBUG:teuthology.orchestra.remote:timed out 2023-12-06T09:04:57.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:04:57.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:04:57.566 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-12-06T09:04:57.917 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:04:57.917 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-390.el9.x86_64"... 2023-12-06T09:04:57.917 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-12-06T09:04:57.931 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-390.el9.x86_64 2023-12-06T09:04:57.932 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-06T09:04:57.932 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T09:04:57.932 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T09:04:58.932 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-06T09:04:58.933 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T09:04:58.991 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2023-12-06T09:04:59.028 DEBUG:teuthology.parallel:result is None 2023-12-06T09:04:59.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T09:04:59.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:04:59.972 DEBUG:teuthology.orchestra.run.smithi132:> true 2023-12-06T09:05:00.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T09:05:00.358 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-390.el9.x86_64"... 2023-12-06T09:05:00.358 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2023-12-06T09:05:00.373 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-390.el9.x86_64 2023-12-06T09:05:00.373 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-06T09:05:00.373 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T09:05:00.373 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T09:05:01.374 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-06T09:05:01.374 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T09:05:01.450 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2023-12-06T09:05:01.484 DEBUG:teuthology.parallel:result is None 2023-12-06T09:05:01.484 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-06T09:05:01.494 INFO:teuthology.task.internal:Creating test directory... 2023-12-06T09:05:01.494 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T09:05:01.497 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T09:05:01.515 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-06T09:05:01.526 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-06T09:05:01.540 INFO:teuthology.task.internal:Creating archive directory... 2023-12-06T09:05:01.541 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T09:05:01.555 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T09:05:01.647 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-06T09:05:01.658 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-06T09:05:01.658 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T09:05:01.661 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T09:05:01.693 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T09:05:01.701 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T09:05:01.707 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T09:05:01.715 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T09:05:01.716 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-06T09:05:01.727 INFO:teuthology.task.internal:Configuring sudo... 2023-12-06T09:05:01.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T09:05:01.752 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T09:05:01.787 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-06T09:05:01.801 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-06T09:05:01.801 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T09:05:01.823 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T09:05:01.844 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T09:05:01.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T09:05:01.975 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T09:05:02.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T09:05:02.106 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:05:02.107 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T09:05:02.175 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T09:05:02.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T09:05:02.272 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T09:05:02.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T09:05:02.398 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:05:02.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T09:05:02.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-12-06T09:05:02.469 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2023-12-06T09:05:02.521 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T09:05:02.555 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T09:05:02.867 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-06T09:05:02.879 INFO:teuthology.task.internal:Starting timer... 2023-12-06T09:05:02.880 INFO:teuthology.run_tasks:Running task pcp... 2023-12-06T09:05:02.895 INFO:teuthology.run_tasks:Running task selinux... 2023-12-06T09:05:02.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2023-12-06T09:05:02.989 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2023-12-06T09:05:02.991 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2023-12-06T09:05:03.043 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2023-12-06T09:05:03.044 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-06T09:05:03.044 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2023-12-06T09:05:03.078 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2023-12-06T09:05:03.079 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2023-12-06T09:05:03.103 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2023-12-06T09:05:03.104 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive'} 2023-12-06T09:05:03.104 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T09:05:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:05:03.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T09:05:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:05:03.175 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-06T09:05:03.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2023-12-06T09:05:03.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2023-12-06T09:05:03.255 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-06T09:05:03.267 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-06T09:05:03.273 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-06T09:05:03.409 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-06T09:05:03.435 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-06T09:05:03.436 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-06T09:10:43.786 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com')] 2023-12-06T09:10:43.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:10:43.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:10:43.874 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-12-06T09:10:43.942 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-12-06T09:10:43.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T09:10:43.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T09:10:44.028 DEBUG:teuthology.orchestra.run.smithi132:> true 2023-12-06T09:10:44.099 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T09:10:44.099 INFO:teuthology.run_tasks:Running task clock... 2023-12-06T09:10:44.112 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-06T09:10:44.112 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T09:10:44.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T09:10:44.116 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T09:10:44.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T09:10:44.147 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T09:10:44.165 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T09:10:44.175 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T09:10:44.192 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T09:10:44.194 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2023-12-06T09:10:44.209 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2023-12-06T09:10:44.222 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2023-12-06T09:10:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T09:10:44.237 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2023-12-06T09:10:44.241 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T09:10:44.253 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T09:10:44.269 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T09:10:44.284 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2023-12-06T09:10:44.287 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T09:10:44.287 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-12-06T09:10:44.287 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.287 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.287 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.287 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.311 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2023-12-06T09:10:44.314 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T09:10:44.314 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2023-12-06T09:10:44.314 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.315 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.315 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.315 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:10:44.315 INFO:teuthology.run_tasks:Running task install... 2023-12-06T09:10:44.325 DEBUG:teuthology.task.install:project ceph 2023-12-06T09:10:44.325 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}} 2023-12-06T09:10:44.325 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'} 2023-12-06T09:10:44.325 INFO:teuthology.task.install:Using flavor: default 2023-12-06T09:10:44.332 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-06T09:10:44.332 INFO:teuthology.task.install:extra packages: [] 2023-12-06T09:10:44.332 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'tag': None, 'wait_for_package': False} 2023-12-06T09:10:44.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T09:10:44.334 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'tag': None, 'wait_for_package': False} 2023-12-06T09:10:44.335 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T09:10:44.475 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/ 2023-12-06T09:10:44.475 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7644.g96fa2578 2023-12-06T09:10:44.481 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/ 2023-12-06T09:10:44.482 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7644.g96fa2578 2023-12-06T09:10:44.586 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T09:10:44.587 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:10:44.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T09:10:44.592 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T09:10:44.592 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:10:44.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T09:10:44.615 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-06T09:10:44.615 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96fa2578c622ca9c169af6f426ca8b1616130e58/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T09:10:44.619 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-06T09:10:44.620 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96fa2578c622ca9c169af6f426ca8b1616130e58/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T09:10:44.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T09:10:44.686 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T09:10:44.761 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T09:10:44.766 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T09:10:44.827 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2023-12-06T09:10:44.830 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2023-12-06T09:10:44.834 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2023-12-06T09:10:44.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2023-12-06T09:10:45.070 INFO:teuthology.orchestra.run.smithi088.stdout:73 files removed 2023-12-06T09:10:45.073 INFO:teuthology.orchestra.run.smithi132.stdout:73 files removed 2023-12-06T09:10:45.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2023-12-06T09:10:45.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2023-12-06T09:10:45.865 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 173 kB/s | 78 kB 00:00 2023-12-06T09:10:45.913 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 158 kB/s | 78 kB 00:00 2023-12-06T09:10:46.291 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-06T09:10:46.313 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 27 kB/s | 9.5 kB 00:00 2023-12-06T09:10:46.708 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2023-12-06T09:10:46.723 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2023-12-06T09:10:47.418 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.4 MB 00:00 2023-12-06T09:10:47.442 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.4 MB 00:00 2023-12-06T09:10:49.002 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2023-12-06T09:10:49.009 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2023-12-06T09:10:52.134 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-12-06T09:10:52.583 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 7.9 MB 00:03 2023-12-06T09:10:54.725 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 17 MB/s | 18 MB 00:01 2023-12-06T09:10:55.885 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 6.8 MB/s | 18 MB 00:02 2023-12-06T09:11:00.826 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.8 MB 00:02 2023-12-06T09:11:01.743 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 5.8 MB 00:02 2023-12-06T09:11:03.127 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 15 kB 00:00 2023-12-06T09:11:03.619 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2023-12-06T09:11:04.572 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-12-06T09:11:09.378 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-12-06T09:11:10.486 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 2.9 MB/s | 20 MB 00:06 2023-12-06T09:11:11.979 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 8.6 M 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.3 M 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T09:11:11.983 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.2 M 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 24 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 718 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 476 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-06T09:11:11.984 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.8 M 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 45 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 129 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 161 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 321 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 297 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 99 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:11.985 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:11:11.986 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:11:11.986 INFO:teuthology.orchestra.run.smithi132.stdout:Install 27 Packages 2023-12-06T09:11:11.986 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2023-12-06T09:11:11.986 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:11.986 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 55 M 2023-12-06T09:11:11.986 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:11:12.909 INFO:teuthology.orchestra.run.smithi132.stdout:(1/29): ceph-base-18.0.0-7644.g96fa2578.el9.x86 6.2 MB/s | 5.2 MB 00:00 2023-12-06T09:11:12.985 INFO:teuthology.orchestra.run.smithi132.stdout:(2/29): ceph-selinux-18.0.0-7644.g96fa2578.el9. 322 kB/s | 24 kB 00:00 2023-12-06T09:11:13.136 INFO:teuthology.orchestra.run.smithi132.stdout:(3/29): libcephfs2-18.0.0-7644.g96fa2578.el9.x8 4.7 MB/s | 718 kB 00:00 2023-12-06T09:11:13.236 INFO:teuthology.orchestra.run.smithi132.stdout:(4/29): libradosstriper1-18.0.0-7644.g96fa2578. 4.6 MB/s | 476 kB 00:00 2023-12-06T09:11:13.419 INFO:teuthology.orchestra.run.smithi132.stdout:(5/29): ceph-common-18.0.0-7644.g96fa2578.el9.x 14 MB/s | 19 MB 00:01 2023-12-06T09:11:13.478 INFO:teuthology.orchestra.run.smithi132.stdout:(6/29): python3-ceph-argparse-18.0.0-7644.g96fa 780 kB/s | 45 kB 00:00 2023-12-06T09:11:13.537 INFO:teuthology.orchestra.run.smithi132.stdout:(7/29): python3-ceph-common-18.0.0-7644.g96fa25 2.2 MB/s | 129 kB 00:00 2023-12-06T09:11:13.597 INFO:teuthology.orchestra.run.smithi132.stdout:(8/29): python3-cephfs-18.0.0-7644.g96fa2578.el 2.7 MB/s | 161 kB 00:00 2023-12-06T09:11:13.654 INFO:teuthology.orchestra.run.smithi132.stdout:(9/29): librgw2-18.0.0-7644.g96fa2578.el9.x86_6 11 MB/s | 4.8 MB 00:00 2023-12-06T09:11:13.679 INFO:teuthology.orchestra.run.smithi132.stdout:(10/29): python3-rados-18.0.0-7644.g96fa2578.el 3.8 MB/s | 321 kB 00:00 2023-12-06T09:11:13.704 INFO:teuthology.orchestra.run.smithi132.stdout:(11/29): python3-rbd-18.0.0-7644.g96fa2578.el9. 5.8 MB/s | 297 kB 00:00 2023-12-06T09:11:13.729 INFO:teuthology.orchestra.run.smithi132.stdout:(12/29): python3-rgw-18.0.0-7644.g96fa2578.el9. 1.9 MB/s | 99 kB 00:00 2023-12-06T09:11:13.871 INFO:teuthology.orchestra.run.smithi132.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 106 kB 00:00 2023-12-06T09:11:13.897 INFO:teuthology.orchestra.run.smithi132.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2023-12-06T09:11:13.931 INFO:teuthology.orchestra.run.smithi132.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 751 kB/s | 44 kB 00:00 2023-12-06T09:11:13.998 INFO:teuthology.orchestra.run.smithi132.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2023-12-06T09:11:14.023 INFO:teuthology.orchestra.run.smithi132.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2023-12-06T09:11:14.040 INFO:teuthology.orchestra.run.smithi132.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 994 kB/s | 41 kB 00:00 2023-12-06T09:11:14.340 INFO:teuthology.orchestra.run.smithi132.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 975 kB/s | 309 kB 00:00 2023-12-06T09:11:14.398 INFO:teuthology.orchestra.run.smithi132.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 418 kB/s | 24 kB 00:00 2023-12-06T09:11:14.532 INFO:teuthology.orchestra.run.smithi132.stdout:(21/29): ceph-radosgw-18.0.0-7644.g96fa2578.el9 3.5 MB/s | 8.6 MB 00:02 2023-12-06T09:11:14.557 INFO:teuthology.orchestra.run.smithi132.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2023-12-06T09:11:14.683 INFO:teuthology.orchestra.run.smithi132.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 6.9 MB/s | 4.4 MB 00:00 2023-12-06T09:11:14.741 INFO:teuthology.orchestra.run.smithi132.stdout:(24/29): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2023-12-06T09:11:14.766 INFO:teuthology.orchestra.run.smithi132.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2023-12-06T09:11:14.792 INFO:teuthology.orchestra.run.smithi132.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-12-06T09:11:15.059 INFO:teuthology.orchestra.run.smithi132.stdout:(27/29): librbd1-18.0.0-7644.g96fa2578.el9.x86_ 11 MB/s | 3.0 MB 00:00 2023-12-06T09:11:15.126 INFO:teuthology.orchestra.run.smithi132.stdout:(28/29): librados2-18.0.0-7644.g96fa2578.el9.x8 9.2 MB/s | 3.3 MB 00:00 2023-12-06T09:11:15.167 INFO:teuthology.orchestra.run.smithi132.stdout:(29/29): thrift-0.15.0-2.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2023-12-06T09:11:15.172 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:11:15.172 INFO:teuthology.orchestra.run.smithi132.stdout:Total 17 MB/s | 55 MB 00:03 2023-12-06T09:11:15.173 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:11:15.202 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-06T09:11:15.208 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:11:15.209 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:11:15.697 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:11:15.697 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:11:17.078 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:11:17.167 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-06T09:11:17.206 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-06T09:11:17.257 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-06T09:11:17.641 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-06T09:11:17.672 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T09:11:17.764 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T09:11:17.778 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:11:17.795 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 8.6 M 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.3 M 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.2 M 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T09:11:17.796 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 24 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 718 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 476 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.8 M 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse 2023-12-06T09:11:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 45 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 129 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 161 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 321 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 297 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 99 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout:Install 27 Packages 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2023-12-06T09:11:17.798 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:17.799 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 55 M 2023-12-06T09:11:17.799 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:11:17.849 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 6/31 2023-12-06T09:11:17.873 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T09:11:17.935 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T09:11:17.964 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-06T09:11:18.001 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 9/31 2023-12-06T09:11:18.067 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T09:11:18.093 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T09:11:18.159 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T09:11:18.265 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-06T09:11:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-06T09:11:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-06T09:11:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-06T09:11:18.411 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-06T09:11:18.444 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-06T09:11:18.755 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-06T09:11:18.784 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T09:11:18.881 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T09:11:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 20/31 2023-12-06T09:11:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-06T09:11:19.400 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-06T09:11:19.429 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T09:11:19.491 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T09:11:19.567 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 24/31 2023-12-06T09:11:19.597 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 25/31 2023-12-06T09:11:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:(1/29): ceph-base-18.0.0-7644.g96fa2578.el9.x86 3.0 MB/s | 5.2 MB 00:01 2023-12-06T09:11:19.745 INFO:teuthology.orchestra.run.smithi088.stdout:(2/29): ceph-selinux-18.0.0-7644.g96fa2578.el9. 362 kB/s | 24 kB 00:00 2023-12-06T09:11:19.897 INFO:teuthology.orchestra.run.smithi088.stdout:(3/29): libcephfs2-18.0.0-7644.g96fa2578.el9.x8 4.6 MB/s | 718 kB 00:00 2023-12-06T09:11:20.030 INFO:teuthology.orchestra.run.smithi088.stdout:(4/29): libradosstriper1-18.0.0-7644.g96fa2578. 3.5 MB/s | 476 kB 00:00 2023-12-06T09:11:20.264 INFO:teuthology.orchestra.run.smithi088.stdout:(5/29): ceph-radosgw-18.0.0-7644.g96fa2578.el9. 3.7 MB/s | 8.6 MB 00:02 2023-12-06T09:11:20.348 INFO:teuthology.orchestra.run.smithi088.stdout:(6/29): python3-ceph-argparse-18.0.0-7644.g96fa 542 kB/s | 45 kB 00:00 2023-12-06T09:11:20.465 INFO:teuthology.orchestra.run.smithi088.stdout:(7/29): python3-ceph-common-18.0.0-7644.g96fa25 1.1 MB/s | 129 kB 00:00 2023-12-06T09:11:20.649 INFO:teuthology.orchestra.run.smithi088.stdout:(8/29): ceph-common-18.0.0-7644.g96fa2578.el9.x 7.0 MB/s | 19 MB 00:02 2023-12-06T09:11:20.674 INFO:teuthology.orchestra.run.smithi088.stdout:(9/29): python3-cephfs-18.0.0-7644.g96fa2578.el 775 kB/s | 161 kB 00:00 2023-12-06T09:11:20.749 INFO:teuthology.orchestra.run.smithi088.stdout:(10/29): librgw2-18.0.0-7644.g96fa2578.el9.x86_ 6.6 MB/s | 4.8 MB 00:00 2023-12-06T09:11:20.774 INFO:teuthology.orchestra.run.smithi088.stdout:(11/29): python3-rbd-18.0.0-7644.g96fa2578.el9. 2.9 MB/s | 297 kB 00:00 2023-12-06T09:11:20.800 INFO:teuthology.orchestra.run.smithi088.stdout:(12/29): python3-rados-18.0.0-7644.g96fa2578.el 2.1 MB/s | 321 kB 00:00 2023-12-06T09:11:20.825 INFO:teuthology.orchestra.run.smithi088.stdout:(13/29): python3-rgw-18.0.0-7644.g96fa2578.el9. 1.3 MB/s | 99 kB 00:00 2023-12-06T09:11:20.967 INFO:teuthology.orchestra.run.smithi088.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2023-12-06T09:11:20.992 INFO:teuthology.orchestra.run.smithi088.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 376 kB/s | 44 kB 00:00 2023-12-06T09:11:21.017 INFO:teuthology.orchestra.run.smithi088.stdout:(16/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 732 kB/s | 159 kB 00:00 2023-12-06T09:11:21.076 INFO:teuthology.orchestra.run.smithi088.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 661 kB 00:00 2023-12-06T09:11:21.101 INFO:teuthology.orchestra.run.smithi088.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2023-12-06T09:11:21.126 INFO:teuthology.orchestra.run.smithi088.stdout:(19/29): python3-prettytable-0.7.2-27.el9.noarc 379 kB/s | 41 kB 00:00 2023-12-06T09:11:21.418 INFO:teuthology.orchestra.run.smithi088.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 83 kB/s | 24 kB 00:00 2023-12-06T09:11:21.527 INFO:teuthology.orchestra.run.smithi088.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2023-12-06T09:11:21.619 INFO:teuthology.orchestra.run.smithi088.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 569 kB/s | 309 kB 00:00 2023-12-06T09:11:21.636 INFO:teuthology.orchestra.run.smithi088.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2023-12-06T09:11:21.715 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T09:11:21.754 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T09:11:21.853 INFO:teuthology.orchestra.run.smithi088.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 884 kB/s | 192 kB 00:00 2023-12-06T09:11:22.321 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T09:11:22.365 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T09:11:22.445 INFO:teuthology.orchestra.run.smithi088.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.0 MB/s | 837 kB 00:00 2023-12-06T09:11:22.446 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T09:11:22.446 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-06T09:11:22.446 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-06T09:11:22.446 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:22.482 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T09:11:22.612 INFO:teuthology.orchestra.run.smithi088.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2023-12-06T09:11:22.920 INFO:teuthology.orchestra.run.smithi088.stdout:(27/29): librados2-18.0.0-7644.g96fa2578.el9.x8 7.0 MB/s | 3.3 MB 00:00 2023-12-06T09:11:23.046 INFO:teuthology.orchestra.run.smithi088.stdout:(28/29): librbd1-18.0.0-7644.g96fa2578.el9.x86_ 7.0 MB/s | 3.0 MB 00:00 2023-12-06T09:11:23.312 INFO:teuthology.orchestra.run.smithi088.stdout:(29/29): libarrow-9.0.0-9.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2023-12-06T09:11:23.319 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:11:23.320 INFO:teuthology.orchestra.run.smithi088.stdout:Total 9.9 MB/s | 55 MB 00:05 2023-12-06T09:11:23.320 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:11:23.355 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:11:23.355 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:11:23.810 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:11:23.810 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:11:25.154 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:11:25.239 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-06T09:11:25.267 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-06T09:11:25.324 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-06T09:11:25.695 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-06T09:11:25.725 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T09:11:25.809 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T09:11:25.894 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 6/31 2023-12-06T09:11:25.934 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T09:11:26.005 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T09:11:26.086 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-06T09:11:26.136 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 9/31 2023-12-06T09:11:26.203 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T09:11:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T09:11:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T09:11:26.396 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-06T09:11:26.433 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-06T09:11:26.473 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-06T09:11:26.514 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-06T09:11:26.568 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-06T09:11:26.600 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-06T09:11:26.904 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-06T09:11:26.939 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T09:11:27.020 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T09:11:27.052 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 20/31 2023-12-06T09:11:27.087 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-06T09:11:27.515 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-06T09:11:27.541 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T09:11:27.613 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T09:11:27.689 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 24/31 2023-12-06T09:11:27.718 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 25/31 2023-12-06T09:11:29.802 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T09:11:29.839 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T09:11:30.391 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T09:11:30.443 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T09:11:30.523 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T09:11:30.523 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-06T09:11:30.523 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-06T09:11:30.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:30.552 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T09:11:31.914 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2023-12-06T09:11:31.915 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:32.733 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T09:11:32.758 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T09:11:32.758 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:11:32.758 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-06T09:11:32.758 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T09:11:32.758 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T09:11:32.758 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:32.760 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T09:11:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T09:11:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 1/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 3/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 4/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 6/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 8/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 9/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T09:11:33.971 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 11/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 12/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 13/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-06T09:11:33.972 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-06T09:11:33.973 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-06T09:11:33.973 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T09:11:33.973 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-06T09:11:33.973 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 30/31 2023-12-06T09:11:34.940 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T09:11:34.940 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:34.940 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2023-12-06T09:11:34.940 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.940 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.940 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.941 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:34.942 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:11:35.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2023-12-06T09:11:35.909 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:26 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:11:36.057 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:11:36.058 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:11:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:11:36.058 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:11:36.058 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:11:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 41 M 2023-12-06T09:11:36.058 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 164 k 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2023-12-06T09:11:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:36.060 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 42 M 2023-12-06T09:11:36.060 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 173 M 2023-12-06T09:11:36.060 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:11:36.248 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 996 kB/s | 186 kB 00:00 2023-12-06T09:11:36.299 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-12-06T09:11:36.357 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-12-06T09:11:36.398 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-06T09:11:36.449 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): libcephsqlite-18.0.0-7644.g96fa2578.el9. 422 kB/s | 164 kB 00:00 2023-12-06T09:11:38.233 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-18.0.0-7644.g96fa2578.el9.x86_ 19 MB/s | 41 MB 00:02 2023-12-06T09:11:38.236 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:11:38.236 INFO:teuthology.orchestra.run.smithi132.stdout:Total 19 MB/s | 42 MB 00:02 2023-12-06T09:11:38.237 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:11:38.256 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:11:38.257 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:11:38.625 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:11:38.625 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:11:39.192 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:11:39.273 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-06T09:11:39.315 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-06T09:11:39.348 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-06T09:11:39.383 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-06T09:11:39.410 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T09:11:39.724 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2023-12-06T09:11:39.725 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:40.532 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T09:11:40.556 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T09:11:40.556 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:11:40.556 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-06T09:11:40.557 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T09:11:40.557 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T09:11:40.557 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:40.558 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T09:11:40.627 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T09:11:40.627 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 1/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 3/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 4/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 6/31 2023-12-06T09:11:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T09:11:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 8/31 2023-12-06T09:11:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 9/31 2023-12-06T09:11:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T09:11:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 11/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 12/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 13/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-06T09:11:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-06T09:11:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-06T09:11:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-06T09:11:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-06T09:11:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-06T09:11:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-06T09:11:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T09:11:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-06T09:11:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 30/31 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-06T09:11:42.733 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:42.734 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:11:43.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2023-12-06T09:11:43.274 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T09:11:43.360 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T09:11:43.678 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:28 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:11:43.823 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:11:43.823 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 41 M 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 164 k 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-06T09:11:43.824 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-06T09:11:43.825 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:43.825 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:11:43.825 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:11:43.825 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2023-12-06T09:11:43.825 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:43.826 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 42 M 2023-12-06T09:11:43.826 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 173 M 2023-12-06T09:11:43.826 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:11:44.023 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 945 kB/s | 186 kB 00:00 2023-12-06T09:11:44.075 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-06T09:11:44.098 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): libcephsqlite-18.0.0-7644.g96fa2578.el9. 601 kB/s | 164 kB 00:00 2023-12-06T09:11:44.132 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-12-06T09:11:44.149 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-06T09:11:45.620 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T09:11:45.677 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 1/6 2023-12-06T09:11:45.677 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2/6 2023-12-06T09:11:45.678 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-06T09:11:45.678 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-06T09:11:45.678 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-06T09:11:45.989 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-06T09:11:45.989 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-16.el9.x86_64 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:45.990 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:11:46.032 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): ceph-test-18.0.0-7644.g96fa2578.el9.x86_ 19 MB/s | 41 MB 00:02 2023-12-06T09:11:46.035 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:11:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 42 MB 00:02 2023-12-06T09:11:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:11:46.055 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:11:46.055 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:11:46.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2023-12-06T09:11:46.413 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:11:46.414 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:11:46.866 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:37 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:11:46.958 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:11:47.021 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:11:47.023 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2023-12-06T09:11:47.023 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 6.6 k 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 2.1 M 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.5 M 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 243 k 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.4 M 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 18 M 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-06T09:11:47.024 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-06T09:11:47.025 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:11:47.026 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2023-12-06T09:11:47.027 INFO:teuthology.orchestra.run.smithi132.stdout:Install 36 Packages 2023-12-06T09:11:47.027 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:47.027 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 30 M 2023-12-06T09:11:47.027 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 111 M 2023-12-06T09:11:47.027 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:11:47.039 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-06T09:11:47.085 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-06T09:11:47.114 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-06T09:11:47.157 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-06T09:11:47.185 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T09:11:47.285 INFO:teuthology.orchestra.run.smithi132.stdout:(1/36): ceph-18.0.0-7644.g96fa2578.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2023-12-06T09:11:47.535 INFO:teuthology.orchestra.run.smithi132.stdout:(2/36): ceph-mgr-18.0.0-7644.g96fa2578.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2023-12-06T09:11:47.577 INFO:teuthology.orchestra.run.smithi132.stdout:(3/36): ceph-mds-18.0.0-7644.g96fa2578.el9.x86_ 4.5 MB/s | 2.1 MB 00:00 2023-12-06T09:11:47.661 INFO:teuthology.orchestra.run.smithi132.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7644.g96fa 2.8 MB/s | 243 kB 00:00 2023-12-06T09:11:47.822 INFO:teuthology.orchestra.run.smithi132.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-12-06T09:11:47.873 INFO:teuthology.orchestra.run.smithi132.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2023-12-06T09:11:47.948 INFO:teuthology.orchestra.run.smithi132.stdout:(7/36): ceph-mon-18.0.0-7644.g96fa2578.el9.x86_ 6.6 MB/s | 4.4 MB 00:00 2023-12-06T09:11:47.973 INFO:teuthology.orchestra.run.smithi132.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 315 kB/s | 32 kB 00:00 2023-12-06T09:11:47.999 INFO:teuthology.orchestra.run.smithi132.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2023-12-06T09:11:48.024 INFO:teuthology.orchestra.run.smithi132.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2023-12-06T09:11:48.057 INFO:teuthology.orchestra.run.smithi132.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-12-06T09:11:48.150 INFO:teuthology.orchestra.run.smithi132.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2023-12-06T09:11:48.174 INFO:teuthology.orchestra.run.smithi132.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 883 kB/s | 103 kB 00:00 2023-12-06T09:11:48.216 INFO:teuthology.orchestra.run.smithi132.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-12-06T09:11:48.242 INFO:teuthology.orchestra.run.smithi132.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 1.6 MB/s | 113 kB 00:00 2023-12-06T09:11:48.268 INFO:teuthology.orchestra.run.smithi132.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-06T09:11:48.493 INFO:teuthology.orchestra.run.smithi132.stdout:(17/36): ceph-osd-18.0.0-7644.g96fa2578.el9.x86 19 MB/s | 18 MB 00:00 2023-12-06T09:11:48.643 INFO:teuthology.orchestra.run.smithi132.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 403 kB/s | 151 kB 00:00 2023-12-06T09:11:48.668 INFO:teuthology.orchestra.run.smithi132.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 250 kB/s | 43 kB 00:00 2023-12-06T09:11:48.693 INFO:teuthology.orchestra.run.smithi132.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 49 kB/s | 22 kB 00:00 2023-12-06T09:11:48.718 INFO:teuthology.orchestra.run.smithi132.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2023-12-06T09:11:48.786 INFO:teuthology.orchestra.run.smithi132.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 416 kB/s | 18 kB 00:00 2023-12-06T09:11:48.811 INFO:teuthology.orchestra.run.smithi132.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2023-12-06T09:11:48.836 INFO:teuthology.orchestra.run.smithi132.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 471 kB/s | 23 kB 00:00 2023-12-06T09:11:48.853 INFO:teuthology.orchestra.run.smithi132.stdout:(25/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2023-12-06T09:11:48.870 INFO:teuthology.orchestra.run.smithi132.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2023-12-06T09:11:48.887 INFO:teuthology.orchestra.run.smithi132.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2023-12-06T09:11:48.904 INFO:teuthology.orchestra.run.smithi132.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 915 kB/s | 46 kB 00:00 2023-12-06T09:11:48.946 INFO:teuthology.orchestra.run.smithi132.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2023-12-06T09:11:48.963 INFO:teuthology.orchestra.run.smithi132.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2023-12-06T09:11:48.988 INFO:teuthology.orchestra.run.smithi132.stdout:(31/36): python3-more-itertools-8.12.0-2.el9.no 670 kB/s | 79 kB 00:00 2023-12-06T09:11:49.014 INFO:teuthology.orchestra.run.smithi132.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2023-12-06T09:11:49.038 INFO:teuthology.orchestra.run.smithi132.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 972 kB/s | 90 kB 00:00 2023-12-06T09:11:49.080 INFO:teuthology.orchestra.run.smithi132.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2023-12-06T09:11:49.106 INFO:teuthology.orchestra.run.smithi132.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2023-12-06T09:11:49.131 INFO:teuthology.orchestra.run.smithi132.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2023-12-06T09:11:49.136 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:11:49.137 INFO:teuthology.orchestra.run.smithi132.stdout:Total 14 MB/s | 30 MB 00:02 2023-12-06T09:11:49.148 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:11:49.187 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:11:49.187 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:11:49.602 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:11:49.603 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:11:50.916 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:11:50.984 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-06T09:11:51.021 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T09:11:51.026 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-06T09:11:51.061 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-06T09:11:51.102 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T09:11:51.130 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-06T09:11:51.168 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-06T09:11:51.201 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-06T09:11:51.292 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-06T09:11:51.370 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-06T09:11:51.412 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-06T09:11:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-06T09:11:51.537 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-06T09:11:51.576 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-06T09:11:51.625 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-06T09:11:51.680 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-06T09:11:51.867 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-06T09:11:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-06T09:11:51.953 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-06T09:11:51.987 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-06T09:11:52.017 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-06T09:11:52.093 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-06T09:11:52.127 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T09:11:52.161 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-06T09:11:52.201 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-06T09:11:52.236 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-06T09:11:52.271 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-06T09:11:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-06T09:11:52.402 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-06T09:11:52.474 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-06T09:11:52.609 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-06T09:11:52.735 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T09:11:52.898 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 31/36 2023-12-06T09:11:52.923 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T09:11:52.948 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T09:11:52.948 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:11:52.949 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-06T09:11:52.949 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T09:11:52.949 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T09:11:52.949 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:53.253 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T09:11:53.253 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 1/6 2023-12-06T09:11:53.253 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2/6 2023-12-06T09:11:53.253 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-06T09:11:53.253 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-06T09:11:53.253 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-06T09:11:53.676 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-06T09:11:53.676 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:53.676 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:11:53.676 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:53.676 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-16.el9.x86_64 2023-12-06T09:11:53.676 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:53.677 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-06T09:11:53.677 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-06T09:11:53.677 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-06T09:11:53.677 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:53.677 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:11:53.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2023-12-06T09:11:54.500 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T09:11:54.527 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T09:11:54.527 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:11:54.527 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-06T09:11:54.527 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T09:11:54.527 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T09:11:54.527 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:54.555 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:39 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:11:54.703 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:11:54.705 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 6.6 k 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 2.1 M 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.5 M 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 243 k 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.4 M 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 18 M 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-06T09:11:54.706 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-06T09:11:54.707 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-06T09:11:54.708 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout:Install 36 Packages 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 30 M 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 111 M 2023-12-06T09:11:54.709 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:11:54.984 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T09:11:55.008 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T09:11:55.008 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:11:55.008 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-06T09:11:55.008 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T09:11:55.008 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T09:11:55.009 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:55.050 INFO:teuthology.orchestra.run.smithi088.stdout:(1/36): ceph-18.0.0-7644.g96fa2578.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2023-12-06T09:11:55.218 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T09:11:55.241 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T09:11:55.241 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:11:55.241 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-06T09:11:55.241 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T09:11:55.241 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T09:11:55.241 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:55.306 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T09:11:55.475 INFO:teuthology.orchestra.run.smithi088.stdout:(2/36): ceph-mgr-18.0.0-7644.g96fa2578.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2023-12-06T09:11:55.542 INFO:teuthology.orchestra.run.smithi088.stdout:(3/36): ceph-mon-18.0.0-7644.g96fa2578.el9.x86_ 8.9 MB/s | 4.4 MB 00:00 2023-12-06T09:11:55.584 INFO:teuthology.orchestra.run.smithi088.stdout:(4/36): ceph-mds-18.0.0-7644.g96fa2578.el9.x86_ 3.0 MB/s | 2.1 MB 00:00 2023-12-06T09:11:55.609 INFO:teuthology.orchestra.run.smithi088.stdout:(5/36): ceph-mgr-modules-core-18.0.0-7644.g96fa 3.5 MB/s | 243 kB 00:00 2023-12-06T09:11:55.777 INFO:teuthology.orchestra.run.smithi088.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 973 kB/s | 187 kB 00:00 2023-12-06T09:11:55.793 INFO:teuthology.orchestra.run.smithi088.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 901 kB/s | 144 kB 00:00 2023-12-06T09:11:55.819 INFO:teuthology.orchestra.run.smithi088.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2023-12-06T09:11:55.844 INFO:teuthology.orchestra.run.smithi088.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2023-12-06T09:11:55.878 INFO:teuthology.orchestra.run.smithi088.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 659 kB/s | 39 kB 00:00 2023-12-06T09:11:55.903 INFO:teuthology.orchestra.run.smithi088.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-12-06T09:11:55.970 INFO:teuthology.orchestra.run.smithi088.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2023-12-06T09:11:55.995 INFO:teuthology.orchestra.run.smithi088.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2023-12-06T09:11:56.021 INFO:teuthology.orchestra.run.smithi088.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-06T09:11:56.046 INFO:teuthology.orchestra.run.smithi088.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-06T09:11:56.071 INFO:teuthology.orchestra.run.smithi088.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-06T09:11:56.188 INFO:teuthology.orchestra.run.smithi088.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2023-12-06T09:11:56.446 INFO:teuthology.orchestra.run.smithi088.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 402 kB/s | 151 kB 00:00 2023-12-06T09:11:56.471 INFO:teuthology.orchestra.run.smithi088.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2023-12-06T09:11:56.555 INFO:teuthology.orchestra.run.smithi088.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2023-12-06T09:11:56.630 INFO:teuthology.orchestra.run.smithi088.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2023-12-06T09:11:56.706 INFO:teuthology.orchestra.run.smithi088.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2023-12-06T09:11:56.731 INFO:teuthology.orchestra.run.smithi088.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2023-12-06T09:11:56.781 INFO:teuthology.orchestra.run.smithi088.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2023-12-06T09:11:56.806 INFO:teuthology.orchestra.run.smithi088.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2023-12-06T09:11:56.857 INFO:teuthology.orchestra.run.smithi088.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2023-12-06T09:11:57.024 INFO:teuthology.orchestra.run.smithi088.stdout:(27/36): ceph-osd-18.0.0-7644.g96fa2578.el9.x86 11 MB/s | 18 MB 00:01 2023-12-06T09:11:57.049 INFO:teuthology.orchestra.run.smithi088.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 191 kB/s | 46 kB 00:00 2023-12-06T09:11:57.074 INFO:teuthology.orchestra.run.smithi088.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 363 kB/s | 79 kB 00:00 2023-12-06T09:11:57.120 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T09:11:57.120 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 1/36 2023-12-06T09:11:57.120 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2/36 2023-12-06T09:11:57.120 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 3/36 2023-12-06T09:11:57.120 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 4/36 2023-12-06T09:11:57.120 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 5/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 6/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-06T09:11:57.121 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-06T09:11:57.122 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-06T09:11:57.123 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-06T09:11:57.124 INFO:teuthology.orchestra.run.smithi088.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2023-12-06T09:11:57.183 INFO:teuthology.orchestra.run.smithi088.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2023-12-06T09:11:57.208 INFO:teuthology.orchestra.run.smithi088.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 428 kB/s | 36 kB 00:00 2023-12-06T09:11:57.284 INFO:teuthology.orchestra.run.smithi088.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-06T09:11:57.359 INFO:teuthology.orchestra.run.smithi088.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2023-12-06T09:11:57.376 INFO:teuthology.orchestra.run.smithi088.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2023-12-06T09:11:57.451 INFO:teuthology.orchestra.run.smithi088.stdout:(36/36): python3-pecan-1.4.2-3.el9.noarch.rpm 638 kB/s | 272 kB 00:00 2023-12-06T09:11:57.457 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:11:57.457 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 30 MB 00:02 2023-12-06T09:11:57.469 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:11:57.508 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:11:57.509 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:11:57.935 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:11:57.935 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:58.171 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-06T09:11:58.172 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-06T09:11:58.173 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-06T09:11:58.174 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-06T09:11:58.174 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-06T09:11:58.174 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-06T09:11:58.174 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-06T09:11:58.174 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:11:58.174 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:11:58.505 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2023-12-06T09:11:59.089 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:50 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:11:59.182 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:11:59.230 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:11:59.231 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:11:59.231 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:11:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:11:59.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2023-12-06T09:11:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-06T09:11:59.358 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-06T09:11:59.390 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-06T09:11:59.460 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-06T09:11:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-06T09:11:59.563 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-06T09:11:59.655 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-06T09:11:59.733 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-06T09:11:59.774 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-06T09:11:59.846 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-06T09:11:59.879 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:50 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:11:59.899 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-06T09:11:59.938 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-06T09:11:59.986 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-06T09:12:00.022 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 778 k 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:00.023 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:00.024 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:00.024 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 778 k 2023-12-06T09:12:00.024 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 790 k 2023-12-06T09:12:00.024 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:00.043 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-06T09:12:00.229 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-06T09:12:00.275 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-06T09:12:00.315 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-06T09:12:00.317 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-18.0.0-7644.g96fa2578.el9.noarch.rpm 2.6 MB/s | 778 kB 00:00 2023-12-06T09:12:00.318 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:00.318 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.6 MB/s | 778 kB 00:00 2023-12-06T09:12:00.318 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:00.324 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:00.324 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:00.330 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:00.330 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:00.349 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-06T09:12:00.378 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-06T09:12:00.401 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:00.455 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-06T09:12:00.489 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T09:12:00.522 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-06T09:12:00.562 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-06T09:12:00.597 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-06T09:12:00.632 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-06T09:12:00.730 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-06T09:12:00.779 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-06T09:12:00.852 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-06T09:12:00.896 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:00.988 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-06T09:12:01.112 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T09:12:01.266 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 31/36 2023-12-06T09:12:01.293 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T09:12:01.318 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T09:12:01.318 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:01.318 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-06T09:12:01.318 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T09:12:01.318 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T09:12:01.318 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:01.710 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:01.710 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:01.710 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:01.710 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:01.710 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:01.710 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:01.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T09:12:02.478 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:53 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:02.625 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:02.625 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:02.625 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:12:02.626 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:02.626 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:02.626 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 141 k 2023-12-06T09:12:02.626 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:02.626 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:02.626 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:02.627 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:02.627 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:02.628 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 141 k 2023-12-06T09:12:02.628 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 400 k 2023-12-06T09:12:02.628 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:02.845 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T09:12:02.872 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T09:12:02.872 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:02.872 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-06T09:12:02.873 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T09:12:02.873 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T09:12:02.873 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:03.073 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-18.0.0-7644.g96fa25 317 kB/s | 141 kB 00:00 2023-12-06T09:12:03.074 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:03.074 INFO:teuthology.orchestra.run.smithi132.stdout:Total 316 kB/s | 141 kB 00:00 2023-12-06T09:12:03.074 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:03.081 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:03.081 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:03.137 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:03.137 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:03.285 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:03.357 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T09:12:03.378 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T09:12:03.381 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T09:12:03.382 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:03.382 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-06T09:12:03.382 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:03.402 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T09:12:03.402 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:03.402 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-06T09:12:03.402 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T09:12:03.402 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T09:12:03.403 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T09:12:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T09:12:03.635 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:03.636 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-06T09:12:03.636 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T09:12:03.636 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T09:12:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T09:12:04.394 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T09:12:04.394 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:04.395 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:04.395 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:04.395 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:04.395 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:04.635 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2023-12-06T09:12:05.228 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:56 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:05.321 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:05.370 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:05.371 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:05.371 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:05.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T09:12:05.608 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T09:12:05.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 1/36 2023-12-06T09:12:05.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2/36 2023-12-06T09:12:05.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 3/36 2023-12-06T09:12:05.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 4/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 5/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 6/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-06T09:12:05.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-06T09:12:05.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-06T09:12:05.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-06T09:12:05.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-06T09:12:05.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-06T09:12:06.019 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:57 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:06.166 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:06.167 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:06.167 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:06.167 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:06.167 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:06.167 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 1.8 M 2023-12-06T09:12:06.167 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 28 k 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 15 k 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout:Install 7 Packages 2023-12-06T09:12:06.168 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:06.169 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.1 M 2023-12-06T09:12:06.169 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 8.7 M 2023-12-06T09:12:06.169 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2023-12-06T09:12:06.503 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-06T09:12:06.504 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:06.505 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:06.519 INFO:teuthology.orchestra.run.smithi132.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7644.g96f 93 kB/s | 28 kB 00:00 2023-12-06T09:12:06.536 INFO:teuthology.orchestra.run.smithi132.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7644.g96fa 48 kB/s | 15 kB 00:00 2023-12-06T09:12:06.578 INFO:teuthology.orchestra.run.smithi132.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7644.g96fa2578 4.9 MB/s | 1.8 MB 00:00 2023-12-06T09:12:06.636 INFO:teuthology.orchestra.run.smithi132.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2023-12-06T09:12:06.720 INFO:teuthology.orchestra.run.smithi132.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 223 kB/s | 41 kB 00:00 2023-12-06T09:12:06.737 INFO:teuthology.orchestra.run.smithi132.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 193 kB/s | 31 kB 00:00 2023-12-06T09:12:06.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2023-12-06T09:12:06.820 INFO:teuthology.orchestra.run.smithi132.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2023-12-06T09:12:06.822 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:06.823 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.2 MB/s | 2.1 MB 00:00 2023-12-06T09:12:06.823 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:06.831 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:06.831 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:06.893 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:06.893 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:07.194 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:07.247 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-06T09:12:07.280 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-06T09:12:07.327 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-06T09:12:07.359 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-06T09:12:07.361 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:52 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:07.389 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 5/7 2023-12-06T09:12:07.454 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:07.503 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:07.504 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:07.505 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:07.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install cephadm 2023-12-06T09:12:07.628 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 6/7 2023-12-06T09:12:07.666 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T09:12:08.143 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:53 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:08.181 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T09:12:08.182 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 1/7 2023-12-06T09:12:08.182 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 2/7 2023-12-06T09:12:08.182 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 3/7 2023-12-06T09:12:08.182 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-06T09:12:08.182 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-06T09:12:08.182 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-06T09:12:08.287 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:08.287 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:08.287 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 778 k 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 778 k 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 790 k 2023-12-06T09:12:08.288 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:08.509 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-06T09:12:08.510 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:08.511 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:08.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T09:12:08.727 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm-18.0.0-7644.g96fa2578.el9.noarch.rpm 1.7 MB/s | 778 kB 00:00 2023-12-06T09:12:08.728 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:08.728 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 778 kB 00:00 2023-12-06T09:12:08.728 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:08.733 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:08.733 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:08.739 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:08.740 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:08.818 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:09.296 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:00 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:09.306 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:09.390 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:09.443 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:09.444 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2023-12-06T09:12:09.444 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 7.4 M 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-06T09:12:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 37 M 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 209 M 2023-12-06T09:12:09.446 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:09.628 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2023-12-06T09:12:09.661 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 531 kB/s | 17 kB 00:00 2023-12-06T09:12:09.695 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-06T09:12:09.763 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:09.772 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-06T09:12:10.004 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-12-06T09:12:10.051 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:10.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:10.051 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:10.051 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:10.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:10.051 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:10.079 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2023-12-06T09:12:10.105 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2023-12-06T09:12:10.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T09:12:10.301 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2023-12-06T09:12:10.488 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2023-12-06T09:12:10.514 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 929 kB/s | 195 kB 00:00 2023-12-06T09:12:10.664 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2023-12-06T09:12:10.829 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:55 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:10.931 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 5.1 MB/s | 7.4 MB 00:01 2023-12-06T09:12:10.933 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:10.934 INFO:teuthology.orchestra.run.smithi132.stdout:Total 25 MB/s | 37 MB 00:01 2023-12-06T09:12:10.946 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:10.970 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:10.971 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:10.977 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 141 k 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:10.978 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:10.979 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:10.979 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 141 k 2023-12-06T09:12:10.979 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 400 k 2023-12-06T09:12:10.979 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:11.207 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-18.0.0-7644.g96fa25 620 kB/s | 141 kB 00:00 2023-12-06T09:12:11.208 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:11.208 INFO:teuthology.orchestra.run.smithi088.stdout:Total 616 kB/s | 141 kB 00:00 2023-12-06T09:12:11.208 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:11.215 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:11.215 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:11.273 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:11.273 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:11.365 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:11.366 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:11.425 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:11.475 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T09:12:11.499 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T09:12:11.499 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:11.499 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-06T09:12:11.500 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:12.065 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-06T09:12:12.168 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-06T09:12:12.269 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-06T09:12:12.305 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-06T09:12:12.466 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T09:12:12.493 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T09:12:12.493 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:12.493 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:12.493 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:12.493 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:12.493 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:12.496 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T09:12:12.583 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-06T09:12:12.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2023-12-06T09:12:13.250 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-06T09:12:13.311 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:58 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:13.367 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-06T09:12:13.405 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:13.454 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:13.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T09:12:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:59 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:14.245 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 1.8 M 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 28 k 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 15 k 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Install 7 Packages 2023-12-06T09:12:14.246 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.1 M 2023-12-06T09:12:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 8.7 M 2023-12-06T09:12:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7644.g96f 157 kB/s | 28 kB 00:00 2023-12-06T09:12:14.569 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-06T09:12:14.594 INFO:teuthology.orchestra.run.smithi088.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7644.g96fa 51 kB/s | 15 kB 00:00 2023-12-06T09:12:14.846 INFO:teuthology.orchestra.run.smithi088.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7644.g96fa2578 3.2 MB/s | 1.8 MB 00:00 2023-12-06T09:12:14.869 INFO:teuthology.orchestra.run.smithi088.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 23 kB/s | 9.0 kB 00:00 2023-12-06T09:12:15.020 INFO:teuthology.orchestra.run.smithi088.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 102 kB/s | 41 kB 00:00 2023-12-06T09:12:15.045 INFO:teuthology.orchestra.run.smithi088.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 154 kB/s | 31 kB 00:00 2023-12-06T09:12:15.478 INFO:teuthology.orchestra.run.smithi088.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 310 kB/s | 188 kB 00:00 2023-12-06T09:12:15.482 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:15.482 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 2.1 MB 00:01 2023-12-06T09:12:15.483 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:15.490 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:15.490 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:15.555 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:15.555 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:15.590 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-06T09:12:15.621 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T09:12:15.851 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:15.913 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-06T09:12:15.947 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-06T09:12:16.002 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-06T09:12:16.034 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-06T09:12:16.064 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 5/7 2023-12-06T09:12:16.304 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 6/7 2023-12-06T09:12:16.332 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T09:12:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T09:12:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 1/7 2023-12-06T09:12:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 2/7 2023-12-06T09:12:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 3/7 2023-12-06T09:12:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-06T09:12:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-06T09:12:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-06T09:12:17.218 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-06T09:12:17.218 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:17.218 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:17.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:17.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:17.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:17.219 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-06T09:12:17.219 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-06T09:12:17.219 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-06T09:12:17.219 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-06T09:12:17.219 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:17.219 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:17.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T09:12:18.013 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:03 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 1/12 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T09:12:18.149 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T09:12:18.150 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-06T09:12:18.150 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-06T09:12:18.150 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-06T09:12:18.150 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-06T09:12:18.150 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-06T09:12:18.158 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:18.159 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 7.4 M 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-06T09:12:18.160 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:18.161 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 37 M 2023-12-06T09:12:18.162 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 209 M 2023-12-06T09:12:18.162 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:18.452 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2023-12-06T09:12:18.494 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 423 kB/s | 17 kB 00:00 2023-12-06T09:12:18.536 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 833 kB/s | 34 kB 00:00 2023-12-06T09:12:18.619 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2023-12-06T09:12:18.630 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-06T09:12:18.630 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:18.630 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:18.630 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:18.630 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:18.631 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:18.820 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-12-06T09:12:18.895 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2023-12-06T09:12:18.920 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-06T09:12:19.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2023-12-06T09:12:19.145 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2023-12-06T09:12:19.337 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2023-12-06T09:12:19.363 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 908 kB/s | 195 kB 00:00 2023-12-06T09:12:19.612 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:10 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:19.758 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 50 k 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-06T09:12:19.759 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:19.760 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:12:19.761 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2023-12-06T09:12:19.761 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:19.761 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.0 M 2023-12-06T09:12:19.761 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2023-12-06T09:12:19.761 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:19.954 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 398 kB/s | 245 kB 00:00 2023-12-06T09:12:20.098 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2023-12-06T09:12:20.124 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 125 kB/s | 24 kB 00:00 2023-12-06T09:12:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 3.9 MB/s | 7.4 MB 00:01 2023-12-06T09:12:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:Total 18 MB/s | 37 MB 00:02 2023-12-06T09:12:20.190 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2023-12-06T09:12:20.197 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:20.216 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2023-12-06T09:12:20.222 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:20.222 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:20.241 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-06T09:12:20.267 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-12-06T09:12:20.283 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): ceph-mgr-rook-18.0.0-7644.g96fa2578.el9 141 kB/s | 50 kB 00:00 2023-12-06T09:12:20.334 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 223 kB/s | 15 kB 00:00 2023-12-06T09:12:20.351 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2023-12-06T09:12:20.409 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2023-12-06T09:12:20.435 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 1.3 MB/s | 197 kB 00:00 2023-12-06T09:12:20.468 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2023-12-06T09:12:20.569 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.4 MB/s | 1.0 MB 00:00 2023-12-06T09:12:20.570 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:20.570 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2023-12-06T09:12:20.570 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:20.583 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:20.583 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:20.604 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:20.605 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:20.678 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:20.679 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:21.195 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:21.364 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-06T09:12:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-06T09:12:21.417 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-06T09:12:21.420 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-06T09:12:21.491 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-06T09:12:21.519 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-06T09:12:21.529 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-06T09:12:21.562 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-06T09:12:21.571 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-06T09:12:21.662 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-06T09:12:21.740 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T09:12:21.773 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-06T09:12:21.777 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T09:12:21.830 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-06T09:12:21.866 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-06T09:12:22.497 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-06T09:12:22.541 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-06T09:12:22.594 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-06T09:12:22.606 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-06T09:12:22.636 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-06T09:12:22.683 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-06T09:12:22.706 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T09:12:23.505 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T09:12:23.505 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 1/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-06T09:12:23.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-06T09:12:23.823 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-06T09:12:23.886 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-06T09:12:23.887 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-06T09:12:23.887 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-06T09:12:23.887 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-06T09:12:23.887 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-06T09:12:23.887 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:23.887 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:24.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T09:12:24.746 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:15 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:24.850 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-06T09:12:24.885 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T09:12:24.893 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 133 k 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-06T09:12:24.894 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-06T09:12:24.895 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:24.895 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:24.895 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:12:24.895 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2023-12-06T09:12:24.895 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:24.895 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.8 M 2023-12-06T09:12:24.896 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 33 M 2023-12-06T09:12:24.896 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:25.172 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.0 MB/s | 228 kB 00:00 2023-12-06T09:12:25.239 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7644.g96fa2578.e 453 kB/s | 133 kB 00:00 2023-12-06T09:12:25.347 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2023-12-06T09:12:25.448 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 280 kB/s | 58 kB 00:00 2023-12-06T09:12:25.473 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 408 kB/s | 51 kB 00:00 2023-12-06T09:12:25.498 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2023-12-06T09:12:25.500 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:25.500 INFO:teuthology.orchestra.run.smithi132.stdout:Total 11 MB/s | 6.8 MB 00:00 2023-12-06T09:12:25.500 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:25.509 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:25.510 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:25.599 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:25.599 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:25.897 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:26.058 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-06T09:12:26.108 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-06T09:12:26.625 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-06T09:12:26.720 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-06T09:12:26.780 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-06T09:12:26.804 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 1/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-06T09:12:27.329 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-06T09:12:27.627 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T09:12:27.628 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/6 2023-12-06T09:12:27.628 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-06T09:12:27.628 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-06T09:12:27.628 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-06T09:12:27.628 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-06T09:12:27.856 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-06T09:12:27.856 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-06T09:12:27.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-06T09:12:27.858 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:27.858 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-06T09:12:27.940 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:27.941 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:28.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2023-12-06T09:12:28.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2023-12-06T09:12:28.774 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:19 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:28.804 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:13 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:28.922 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:28.922 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 860 k 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2023-12-06T09:12:28.923 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:28.924 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 939 k 2023-12-06T09:12:28.924 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.7 M 2023-12-06T09:12:28.924 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:28.949 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:28.950 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:12:28.950 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:28.950 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:12:28.950 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:28.950 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 50 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:28.951 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:12:28.952 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2023-12-06T09:12:28.952 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:28.952 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.0 M 2023-12-06T09:12:28.952 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2023-12-06T09:12:28.952 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:29.067 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 551 kB/s | 79 kB 00:00 2023-12-06T09:12:29.244 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 162 kB/s | 24 kB 00:00 2023-12-06T09:12:29.269 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 103 kB/s | 18 kB 00:00 2023-12-06T09:12:29.352 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2023-12-06T09:12:29.358 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-fuse-18.0.0-7644.g96fa2578.el9.x86_ 1.9 MB/s | 860 kB 00:00 2023-12-06T09:12:29.360 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:29.360 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.1 MB/s | 939 kB 00:00 2023-12-06T09:12:29.360 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:29.369 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2023-12-06T09:12:29.381 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:29.381 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:29.395 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): ceph-mgr-rook-18.0.0-7644.g96fa2578.el9 167 kB/s | 50 kB 00:00 2023-12-06T09:12:29.422 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2023-12-06T09:12:29.445 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 567 kB/s | 43 kB 00:00 2023-12-06T09:12:29.456 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:29.456 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:29.645 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2023-12-06T09:12:29.647 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:29.670 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2023-12-06T09:12:29.788 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 577 kB/s | 197 kB 00:00 2023-12-06T09:12:29.792 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-06T09:12:29.813 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2023-12-06T09:12:29.855 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2023-12-06T09:12:29.875 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T09:12:29.955 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2023-12-06T09:12:29.957 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:29.957 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-12-06T09:12:29.957 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:29.971 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:29.971 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:30.069 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:30.070 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:30.537 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T09:12:30.537 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T09:12:30.586 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:30.751 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-06T09:12:30.806 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-06T09:12:30.861 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-06T09:12:30.877 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-06T09:12:30.877 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:30.877 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:30.877 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-06T09:12:30.877 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:30.877 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:30.898 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-06T09:12:30.940 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-06T09:12:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-06T09:12:31.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-volume 2023-12-06T09:12:31.150 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-06T09:12:31.200 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-06T09:12:31.712 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:22 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:31.858 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:31.858 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 262 k 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 262 k 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.2 M 2023-12-06T09:12:31.859 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:31.925 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-06T09:12:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-06T09:12:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-06T09:12:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-06T09:12:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T09:12:32.555 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-volume-18.0.0-7644.g96fa2578.el9.noarch.rp 377 kB/s | 262 kB 00:00 2023-12-06T09:12:32.556 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:32.556 INFO:teuthology.orchestra.run.smithi132.stdout:Total 376 kB/s | 262 kB 00:00 2023-12-06T09:12:32.556 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:32.564 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:32.564 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:32.600 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:32.601 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:32.811 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:32.874 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:32.889 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:32.889 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:32.889 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-06T09:12:32.889 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:32.913 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T09:12:32.913 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 1/13 2023-12-06T09:12:32.913 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-06T09:12:32.913 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-06T09:12:32.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-06T09:12:33.300 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-06T09:12:33.300 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:33.300 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:33.301 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:33.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T09:12:33.903 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:33.903 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:33.903 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:33.903 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:33.903 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:33.904 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:34.142 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:19 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:34.247 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2023-12-06T09:12:34.286 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 133 k 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-06T09:12:34.287 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 6.8 M 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 33 M 2023-12-06T09:12:34.288 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:34.566 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.0 MB/s | 228 kB 00:00 2023-12-06T09:12:34.716 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7644.g96fa2578.e 361 kB/s | 133 kB 00:00 2023-12-06T09:12:34.792 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2023-12-06T09:12:34.856 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:25 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:34.892 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2023-12-06T09:12:34.917 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 289 kB/s | 58 kB 00:00 2023-12-06T09:12:34.942 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2023-12-06T09:12:34.944 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:34.945 INFO:teuthology.orchestra.run.smithi088.stdout:Total 10 MB/s | 6.8 MB 00:00 2023-12-06T09:12:34.945 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:34.953 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:34.953 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:35.001 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:35.001 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 126 k 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:35.002 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:35.003 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 126 k 2023-12-06T09:12:35.003 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 456 k 2023-12-06T09:12:35.003 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:35.041 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:35.041 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:35.228 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-18.0.0-7644.g96fa2578.el9.x86_64 563 kB/s | 126 kB 00:00 2023-12-06T09:12:35.228 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:35.229 INFO:teuthology.orchestra.run.smithi132.stdout:Total 559 kB/s | 126 kB 00:00 2023-12-06T09:12:35.229 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:35.234 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:35.234 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:35.290 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:35.290 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:35.338 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:35.508 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-06T09:12:35.543 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:35.561 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-06T09:12:35.988 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:36.078 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-06T09:12:36.162 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-06T09:12:36.223 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-06T09:12:36.248 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T09:12:36.268 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:36.269 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:36.269 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:36.269 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:36.269 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:36.269 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:36.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2023-12-06T09:12:37.069 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T09:12:37.069 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/6 2023-12-06T09:12:37.070 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-06T09:12:37.070 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-06T09:12:37.070 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-06T09:12:37.070 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-06T09:12:37.129 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:28 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:37.226 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:37.277 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:37.278 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:37.278 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:37.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2023-12-06T09:12:37.395 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-06T09:12:37.395 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:37.395 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:37.396 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:37.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2023-12-06T09:12:37.946 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:28 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:38.092 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 31 k 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:38.093 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:38.094 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:38.094 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:38.094 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 k 2023-12-06T09:12:38.094 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 139 k 2023-12-06T09:12:38.094 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:38.226 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:23 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:38.248 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-18.0.0-7644.g96fa2578.el9.x86_6 202 kB/s | 31 kB 00:00 2023-12-06T09:12:38.249 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:38.249 INFO:teuthology.orchestra.run.smithi132.stdout:Total 200 kB/s | 31 kB 00:00 2023-12-06T09:12:38.249 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:38.252 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:38.252 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:38.292 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:38.292 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:38.372 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 860 k 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:38.373 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2023-12-06T09:12:38.374 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:38.374 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 939 k 2023-12-06T09:12:38.374 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.7 M 2023-12-06T09:12:38.374 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:38.375 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:38.479 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:38.544 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 465 kB/s | 79 kB 00:00 2023-12-06T09:12:38.702 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-fuse-18.0.0-7644.g96fa2578.el9.x86_ 2.6 MB/s | 860 kB 00:00 2023-12-06T09:12:38.704 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:38.704 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.8 MB/s | 939 kB 00:00 2023-12-06T09:12:38.704 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:38.724 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:38.725 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:38.799 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:38.800 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:38.862 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:38.985 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:39.110 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-06T09:12:39.129 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:39.130 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:39.130 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:39.130 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:39.130 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:39.130 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:39.186 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T09:12:39.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2023-12-06T09:12:39.905 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T09:12:39.905 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T09:12:39.912 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:30 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:40.009 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:40.060 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:40.061 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:40.061 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:40.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2023-12-06T09:12:40.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-06T09:12:40.242 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:40.242 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:40.242 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-06T09:12:40.242 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:40.242 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:40.501 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-volume 2023-12-06T09:12:40.718 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:31 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:40.814 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:40.864 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:40.865 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:40.865 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:40.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2023-12-06T09:12:41.102 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:26 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:41.246 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 262 k 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:41.247 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:41.248 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 262 k 2023-12-06T09:12:41.248 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.2 M 2023-12-06T09:12:41.248 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:41.498 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-volume-18.0.0-7644.g96fa2578.el9.noarch.rp 1.0 MB/s | 262 kB 00:00 2023-12-06T09:12:41.499 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:41.499 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-12-06T09:12:41.499 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:41.507 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:41.507 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:41.528 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:32 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:41.541 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:41.541 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:41.622 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:41.673 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:41.674 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:41.674 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:41.744 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2023-12-06T09:12:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:41.833 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:41.833 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-06T09:12:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:42.329 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:33 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:42.424 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:42.475 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:42.476 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:42.476 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:42.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2023-12-06T09:12:42.958 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T09:12:42.958 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:42.958 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:42.958 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T09:12:42.959 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:42.959 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:43.146 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:34 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:43.240 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:43.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2023-12-06T09:12:43.291 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:43.292 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:43.292 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:43.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2023-12-06T09:12:43.855 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:28 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:43.956 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:34 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:43.999 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:44.000 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:44.000 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:12:44.000 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:44.000 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:44.000 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 126 k 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 126 k 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 456 k 2023-12-06T09:12:44.001 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:44.051 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:44.101 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:44.102 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:12:44.102 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:44.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2023-12-06T09:12:44.220 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-18.0.0-7644.g96fa2578.el9.x86_64 580 kB/s | 126 kB 00:00 2023-12-06T09:12:44.220 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:44.220 INFO:teuthology.orchestra.run.smithi088.stdout:Total 576 kB/s | 126 kB 00:00 2023-12-06T09:12:44.221 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:44.226 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:44.226 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:44.281 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:44.281 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:44.423 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:44.535 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:44.763 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:35 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:44.910 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 85 k 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:44.911 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:44.912 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:44.912 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:44.912 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:44.912 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 85 k 2023-12-06T09:12:44.912 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 231 k 2023-12-06T09:12:44.912 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:45.008 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:45.143 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-18.0.0-7644.g96fa2578.el9.x86_64.rpm 369 kB/s | 85 kB 00:00 2023-12-06T09:12:45.144 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:45.144 INFO:teuthology.orchestra.run.smithi132.stdout:Total 367 kB/s | 85 kB 00:00 2023-12-06T09:12:45.144 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:45.151 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:45.151 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:45.206 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:45.206 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:45.335 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:45.335 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:45.336 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:45.336 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:45.336 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:45.336 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:45.361 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:45.476 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:45.596 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2023-12-06T09:12:45.915 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:46.195 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:31 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:46.245 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:46.245 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:46.245 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:46.245 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:46.245 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:46.245 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:46.291 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:46.341 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:46.342 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:46.342 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:46.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2023-12-06T09:12:46.465 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2023-12-06T09:12:46.992 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:32 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:47.063 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:38 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:47.136 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 31 k 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:47.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:47.138 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 31 k 2023-12-06T09:12:47.138 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 139 k 2023-12-06T09:12:47.138 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:47.214 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:47.253 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.0 M 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 12 M 2023-12-06T09:12:47.254 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:47.397 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-18.0.0-7644.g96fa2578.el9.x86_6 120 kB/s | 31 kB 00:00 2023-12-06T09:12:47.398 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:47.398 INFO:teuthology.orchestra.run.smithi088.stdout:Total 119 kB/s | 31 kB 00:00 2023-12-06T09:12:47.398 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:47.402 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:47.402 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:47.422 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:47.423 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:47.513 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:47.645 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:47.763 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-18.0.0-7644.g96fa2578.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2023-12-06T09:12:47.763 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:47.764 INFO:teuthology.orchestra.run.smithi132.stdout:Total 5.5 MB/s | 3.0 MB 00:00 2023-12-06T09:12:47.764 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:47.773 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:47.773 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:47.847 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:47.847 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:48.266 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:48.306 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:48.307 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:48.307 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:48.307 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:48.307 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:48.307 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:48.339 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:48.364 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:48.364 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-06T09:12:48.364 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T09:12:48.364 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T09:12:48.364 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:48.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2023-12-06T09:12:49.049 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:34 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:49.144 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:49.194 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:49.195 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:49.195 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:49.251 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2023-12-06T09:12:49.503 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:49.503 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:49.503 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:49.503 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:49.503 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:49.503 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:49.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2023-12-06T09:12:49.823 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:34 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:49.917 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:49.967 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:49.968 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:49.968 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:50.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados 2023-12-06T09:12:50.341 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:41 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:50.487 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 171 k 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:12:50.488 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:50.489 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 171 k 2023-12-06T09:12:50.489 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 494 k 2023-12-06T09:12:50.489 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:50.617 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:35 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:50.711 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:50.761 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:50.762 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:50.762 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:50.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw 2023-12-06T09:12:50.892 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-18.0.0-7644.g96fa2578.el9.x86_64.rpm 424 kB/s | 171 kB 00:00 2023-12-06T09:12:50.892 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:50.893 INFO:teuthology.orchestra.run.smithi132.stdout:Total 423 kB/s | 171 kB 00:00 2023-12-06T09:12:50.893 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:50.901 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:50.901 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:50.957 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:50.957 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:51.109 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:12:51.214 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:51.400 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:36 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:51.493 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:51.543 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:51.544 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:51.544 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:51.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs 2023-12-06T09:12:51.649 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:51.943 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:51.943 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:51.943 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:12:51.943 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:51.943 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:51.943 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:12:52.189 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:37 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:52.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base-debuginfo 2023-12-06T09:12:52.282 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:52.332 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:52.333 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:52.334 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:52.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd 2023-12-06T09:12:52.791 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:43 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:12:52.941 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:12:52.941 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:52.941 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 73 M 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 21 M 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2023-12-06T09:12:52.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:12:52.943 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 94 M 2023-12-06T09:12:52.943 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 508 M 2023-12-06T09:12:52.943 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:12:52.975 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:37 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:53.069 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:12:53.119 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:53.120 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:12:53.120 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:53.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2023-12-06T09:12:53.786 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:38 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:53.932 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:53.932 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:53.932 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2023-12-06T09:12:53.932 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:53.932 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:53.932 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 85 k 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 85 k 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 231 k 2023-12-06T09:12:53.933 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:54.392 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-18.0.0-7644.g96fa2578.el9.x86_64.rpm 186 kB/s | 85 kB 00:00 2023-12-06T09:12:54.392 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:54.392 INFO:teuthology.orchestra.run.smithi088.stdout:Total 186 kB/s | 85 kB 00:00 2023-12-06T09:12:54.393 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:54.399 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:54.400 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:54.455 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:54.455 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:54.598 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:54.681 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:55.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:55.398 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:55.398 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:55.398 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:55.398 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:55.398 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:12:55.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2023-12-06T09:12:56.182 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:41 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:12:56.331 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:12:56.332 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:56.332 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:12:56.332 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:56.332 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.0 M 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 12 M 2023-12-06T09:12:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:12:58.177 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-18.0.0-7644.g96fa2578.el9.x86_64.rpm 1.6 MB/s | 3.0 MB 00:01 2023-12-06T09:12:58.178 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:58.178 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.6 MB/s | 3.0 MB 00:01 2023-12-06T09:12:58.179 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:12:58.189 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:12:58.190 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:12:58.266 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:12:58.266 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:12:58.681 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:12:58.728 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): ceph-base-debuginfo-18.0.0-7644.g96fa257 13 MB/s | 73 MB 00:05 2023-12-06T09:12:58.736 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:58.761 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:58.761 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T09:12:58.761 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-06T09:12:58.761 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T09:12:58.761 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T09:12:58.761 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:58.937 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-debuginfo-18.0.0-7644.g96fa2578.el9 3.4 MB/s | 21 MB 00:05 2023-12-06T09:12:58.939 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:12:58.939 INFO:teuthology.orchestra.run.smithi132.stdout:Total 16 MB/s | 94 MB 00:05 2023-12-06T09:12:58.939 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:12:58.944 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:12:58.944 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:12:59.553 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:12:59.553 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:12:59.948 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:12:59.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:59.948 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:12:59.948 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:12:59.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:12:59.948 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:13:00.206 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2023-12-06T09:13:00.798 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:45 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:13:00.944 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:13:00.944 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 171 k 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 171 k 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 494 k 2023-12-06T09:13:00.945 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:13:01.305 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-18.0.0-7644.g96fa2578.el9.x86_64.rpm 476 kB/s | 171 kB 00:00 2023-12-06T09:13:01.305 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:13:01.305 INFO:teuthology.orchestra.run.smithi088.stdout:Total 474 kB/s | 171 kB 00:00 2023-12-06T09:13:01.305 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:13:01.313 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:13:01.313 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:13:01.368 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:13:01.369 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:13:01.523 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:13:01.621 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:13:02.096 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:13:02.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:13:02.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:02.415 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:13:02.415 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:13:02.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:02.415 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:13:02.492 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:13:02.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base-debuginfo 2023-12-06T09:13:03.246 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:48 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:13:03.403 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:13:03.404 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:03.404 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:13:03.404 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:03.404 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:13:03.404 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 73 M 2023-12-06T09:13:03.404 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 21 M 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 94 M 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 508 M 2023-12-06T09:13:03.405 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:13:04.649 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): ceph-debuginfo-18.0.0-7644.g96fa2578.el9 17 MB/s | 21 MB 00:01 2023-12-06T09:13:07.824 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-base-debuginfo-18.0.0-7644.g96fa257 17 MB/s | 73 MB 00:04 2023-12-06T09:13:07.826 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:13:07.826 INFO:teuthology.orchestra.run.smithi088.stdout:Total 21 MB/s | 94 MB 00:04 2023-12-06T09:13:07.827 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:13:07.831 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:13:07.831 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:13:08.463 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:13:08.463 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:13:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T09:13:10.179 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T09:13:11.370 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:13:13.764 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T09:13:13.764 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/2 2023-12-06T09:13:14.040 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T09:13:14.041 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:13:14.041 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:13:14.041 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:13:14.041 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:13:14.041 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:13:14.041 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:13:14.290 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-common-debuginfo 2023-12-06T09:13:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:05 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:13:15.029 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 404 M 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:13:15.030 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:13:15.031 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:13:15.031 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:13:15.031 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 404 M 2023-12-06T09:13:15.031 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.6 G 2023-12-06T09:13:15.031 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:13:18.918 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T09:13:18.989 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T09:13:22.428 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T09:13:22.428 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/2 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:22.717 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:13:22.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-common-debuginfo 2023-12-06T09:13:23.533 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:08 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:13:23.677 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:13:23.678 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:23.678 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:13:23.678 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:23.678 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:13:23.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 404 M 2023-12-06T09:13:23.678 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 404 M 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.6 G 2023-12-06T09:13:23.679 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:13:41.498 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-common-debuginfo-18.0.0-7644.g96fa2578.el9 15 MB/s | 404 MB 00:26 2023-12-06T09:13:41.498 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:13:41.499 INFO:teuthology.orchestra.run.smithi132.stdout:Total 15 MB/s | 404 MB 00:26 2023-12-06T09:13:41.499 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:13:41.503 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:13:41.504 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:13:43.902 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:13:43.902 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:13:52.781 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-common-debuginfo-18.0.0-7644.g96fa2578.el9 14 MB/s | 404 MB 00:29 2023-12-06T09:13:52.782 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:13:52.782 INFO:teuthology.orchestra.run.smithi088.stdout:Total 14 MB/s | 404 MB 00:29 2023-12-06T09:13:52.783 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:13:52.787 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:13:52.787 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:13:55.189 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:13:55.189 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:14:24.950 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:14:25.244 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:14:31.793 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:14:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:14:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:32.060 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:14:32.060 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:14:32.061 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:32.061 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:14:32.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-06T09:14:32.930 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:23 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:14:33.087 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.4 M 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.4 M 2023-12-06T09:14:33.088 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 6.2 M 2023-12-06T09:14:33.089 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:14:33.664 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-76 2.4 MB/s | 1.4 MB 00:00 2023-12-06T09:14:33.665 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:14:33.665 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.4 MB/s | 1.4 MB 00:00 2023-12-06T09:14:33.665 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:14:33.669 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:14:33.669 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:14:33.711 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:14:33.712 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:14:33.961 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:14:34.054 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T09:14:34.480 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T09:14:34.763 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T09:14:34.763 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:34.763 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:14:34.763 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:14:34.763 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:34.763 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:14:34.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-06T09:14:35.524 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:26 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:14:35.672 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 172 M 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:14:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:14:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:14:35.674 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 172 M 2023-12-06T09:14:35.674 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 730 M 2023-12-06T09:14:35.674 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:14:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:14:36.558 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:14:44.227 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:14:44.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:14:44.506 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:44.506 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:14:44.506 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:14:44.507 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:44.507 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:14:44.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-06T09:14:45.334 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:30 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:14:45.491 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:14:45.492 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:14:45.492 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:14:45.492 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:14:45.492 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:14:45.492 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.4 M 2023-12-06T09:14:45.492 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.4 M 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 6.2 M 2023-12-06T09:14:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:14:46.010 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-76 2.7 MB/s | 1.4 MB 00:00 2023-12-06T09:14:46.010 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:14:46.010 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.7 MB/s | 1.4 MB 00:00 2023-12-06T09:14:46.011 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:14:46.014 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:14:46.014 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:14:46.057 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:14:46.057 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:14:46.314 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:14:46.401 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T09:14:46.830 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T09:14:47.109 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T09:14:47.109 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:47.109 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:14:47.110 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:14:47.110 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:47.110 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:14:47.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-06T09:14:47.852 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:32 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:14:47.998 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 172 M 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 172 M 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 730 M 2023-12-06T09:14:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:14:51.216 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-radosgw-debuginfo-18.0.0-7644.g96fa2578.el 11 MB/s | 172 MB 00:15 2023-12-06T09:14:51.217 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:14:51.217 INFO:teuthology.orchestra.run.smithi132.stdout:Total 11 MB/s | 172 MB 00:15 2023-12-06T09:14:51.217 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:14:51.221 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:14:51.221 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:14:52.262 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:14:52.262 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:14:57.670 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-radosgw-debuginfo-18.0.0-7644.g96fa2578.el 18 MB/s | 172 MB 00:09 2023-12-06T09:14:57.671 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:14:57.671 INFO:teuthology.orchestra.run.smithi088.stdout:Total 18 MB/s | 172 MB 00:09 2023-12-06T09:14:57.671 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:14:57.675 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:14:57.676 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:14:58.718 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:14:58.719 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:15:09.952 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:15:10.145 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T09:15:14.827 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T09:15:15.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T09:15:15.104 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:15:15.104 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:15:15.104 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:15:15.104 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:15:15.104 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:15:15.305 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test-debuginfo 2023-12-06T09:15:15.889 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:06 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:15:16.035 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:15:16.035 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:15:16.035 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 622 M 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:15:16.036 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:15:16.037 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 622 M 2023-12-06T09:15:16.037 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.8 G 2023-12-06T09:15:16.037 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:15:16.319 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:15:16.414 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T09:15:21.015 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T09:15:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T09:15:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:15:21.310 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:15:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:15:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:15:21.310 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:15:21.560 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test-debuginfo 2023-12-06T09:15:22.137 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:04:07 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:15:22.282 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 622 M 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:15:22.283 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 622 M 2023-12-06T09:15:22.284 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.8 G 2023-12-06T09:15:22.284 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:15:54.071 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-test-debuginfo-18.0.0-7644.g96fa2578.el9.x 16 MB/s | 622 MB 00:38 2023-12-06T09:15:54.072 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:15:54.072 INFO:teuthology.orchestra.run.smithi132.stdout:Total 16 MB/s | 622 MB 00:38 2023-12-06T09:15:54.072 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:15:54.077 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:15:54.077 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:15:57.767 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:15:57.767 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:16:03.061 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-test-debuginfo-18.0.0-7644.g96fa2578.el9.x 15 MB/s | 622 MB 00:40 2023-12-06T09:16:03.062 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:16:03.063 INFO:teuthology.orchestra.run.smithi088.stdout:Total 15 MB/s | 622 MB 00:40 2023-12-06T09:16:03.063 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:16:03.068 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:16:03.068 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:16:06.730 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:16:06.730 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:17:03.244 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:17:03.453 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:17:11.720 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:17:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:17:12.917 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:17:13.194 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:17:13.194 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:13.195 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:17:13.195 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:13.195 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:13.195 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:17:13.486 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base-debuginfo 2023-12-06T09:17:14.069 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:06:05 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:17:14.164 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:17:14.214 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:17:14.215 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T09:17:14.215 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:17:14.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-debuginfo 2023-12-06T09:17:14.866 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:06:05 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:17:15.013 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:17:15.014 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:15.015 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 19 M 2023-12-06T09:17:15.015 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 100 M 2023-12-06T09:17:15.015 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:17:16.149 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-mgr-debuginfo-18.0.0-7644.g96fa2578.el9.x8 17 MB/s | 19 MB 00:01 2023-12-06T09:17:16.150 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:17:16.150 INFO:teuthology.orchestra.run.smithi132.stdout:Total 17 MB/s | 19 MB 00:01 2023-12-06T09:17:16.150 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:17:16.154 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:17:16.155 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:17:16.299 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:17:16.299 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:17:18.482 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:17:18.654 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:19.631 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:19.903 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:19.903 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:19.903 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:17:19.903 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:19.903 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:19.903 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:17:20.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mds-debuginfo 2023-12-06T09:17:20.186 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:17:20.473 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:17:20.474 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:20.474 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:17:20.474 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:20.474 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:20.474 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:17:20.697 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:06:11 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:17:20.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base-debuginfo 2023-12-06T09:17:20.843 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 47 M 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:17:20.844 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:20.845 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:17:20.845 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:20.845 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 47 M 2023-12-06T09:17:20.845 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 168 M 2023-12-06T09:17:20.845 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:17:21.335 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:06 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:17:21.429 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T09:17:21.480 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:17:21.481 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-12-06T09:17:21.481 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:17:21.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-debuginfo 2023-12-06T09:17:22.118 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:07 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:17:22.263 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:17:22.263 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:22.263 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:17:22.263 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:22.263 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 19 M 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 100 M 2023-12-06T09:17:22.264 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:17:23.576 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-mds-debuginfo-18.0.0-7644.g96fa2578.el9.x8 17 MB/s | 47 MB 00:02 2023-12-06T09:17:23.577 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:17:23.577 INFO:teuthology.orchestra.run.smithi132.stdout:Total 17 MB/s | 47 MB 00:02 2023-12-06T09:17:23.578 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:17:23.582 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:17:23.582 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:17:23.892 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:17:23.892 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:17:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mgr-debuginfo-18.0.0-7644.g96fa2578.el9.x8 11 MB/s | 19 MB 00:01 2023-12-06T09:17:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:17:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 19 MB 00:01 2023-12-06T09:17:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:17:23.998 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:17:23.998 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:17:24.144 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:17:24.144 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:17:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:17:26.398 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:27.662 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:27.662 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:27.663 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:17:27.663 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:27.663 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:27.663 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:17:27.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mds-debuginfo 2023-12-06T09:17:28.463 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:13 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:17:28.565 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:17:28.618 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 47 M 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:17:28.619 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:28.620 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:17:28.620 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:28.620 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 47 M 2023-12-06T09:17:28.620 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 168 M 2023-12-06T09:17:28.620 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:17:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:30.005 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:30.267 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:30.267 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:30.267 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:17:30.268 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:30.268 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:30.268 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:17:30.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mon-debuginfo 2023-12-06T09:17:31.042 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:06:22 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:17:31.190 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:17:31.190 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 111 M 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 111 M 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 507 M 2023-12-06T09:17:31.191 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:17:32.403 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mds-debuginfo-18.0.0-7644.g96fa2578.el9.x8 12 MB/s | 47 MB 00:03 2023-12-06T09:17:32.403 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:17:32.404 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 47 MB 00:03 2023-12-06T09:17:32.404 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:17:32.408 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:17:32.408 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:17:32.717 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:17:32.718 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:17:37.317 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:17:37.433 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:38.111 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-mon-debuginfo-18.0.0-7644.g96fa2578.el9.x8 16 MB/s | 111 MB 00:06 2023-12-06T09:17:38.112 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:17:38.113 INFO:teuthology.orchestra.run.smithi132.stdout:Total 16 MB/s | 111 MB 00:06 2023-12-06T09:17:38.113 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:17:38.117 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:17:38.117 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:17:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:17:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:17:39.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:39.018 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:39.018 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:17:39.018 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:39.018 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:39.018 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:17:39.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mon-debuginfo 2023-12-06T09:17:39.776 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:24 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:17:39.919 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:17:39.920 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:39.920 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:17:39.920 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:39.920 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 111 M 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 111 M 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 507 M 2023-12-06T09:17:39.921 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:17:45.803 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mon-debuginfo-18.0.0-7644.g96fa2578.el9.x8 19 MB/s | 111 MB 00:05 2023-12-06T09:17:45.804 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:17:45.804 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 111 MB 00:05 2023-12-06T09:17:45.805 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:17:45.809 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:17:45.809 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:17:46.495 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:17:46.495 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:17:50.447 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:17:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:54.179 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:17:54.180 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:54.180 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:17:54.180 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:17:54.180 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:54.180 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:17:54.407 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-osd-debuginfo 2023-12-06T09:17:54.992 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:06:46 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:17:55.137 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:17:55.137 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:55.137 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:17:55.137 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:55.137 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 311 M 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 311 M 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.3 G 2023-12-06T09:17:55.138 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:17:58.120 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:17:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:18:01.495 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:18:01.772 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:18:01.772 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:18:01.772 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:18:01.772 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:18:01.772 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:18:01.772 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:18:01.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-osd-debuginfo 2023-12-06T09:18:02.537 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:47 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:18:02.683 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 311 M 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:18:02.684 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:18:02.685 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 311 M 2023-12-06T09:18:02.685 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.3 G 2023-12-06T09:18:02.685 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:18:12.311 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-osd-debuginfo-18.0.0-7644.g96fa2578.el9.x8 18 MB/s | 311 MB 00:17 2023-12-06T09:18:12.312 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:18:12.312 INFO:teuthology.orchestra.run.smithi132.stdout:Total 18 MB/s | 311 MB 00:17 2023-12-06T09:18:12.312 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:18:12.316 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:18:12.316 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:18:14.162 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:18:14.163 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:18:28.674 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-osd-debuginfo-18.0.0-7644.g96fa2578.el9.x8 12 MB/s | 311 MB 00:25 2023-12-06T09:18:28.674 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:18:28.675 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 311 MB 00:25 2023-12-06T09:18:28.675 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:18:28.679 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:18:28.679 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:18:30.528 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:18:30.528 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:18:45.950 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:18:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:18:54.381 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:18:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:18:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:18:54.675 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:18:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:18:54.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:18:54.675 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:18:54.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse-debuginfo 2023-12-06T09:18:55.488 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:07:46 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.8 M 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:18:55.635 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:18:55.636 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:18:55.636 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:18:55.636 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 5.8 M 2023-12-06T09:18:55.636 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 28 M 2023-12-06T09:18:55.636 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:18:58.586 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x 2.0 MB/s | 5.8 MB 00:02 2023-12-06T09:18:58.586 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:18:58.586 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.0 MB/s | 5.8 MB 00:02 2023-12-06T09:18:58.586 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:18:58.590 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:18:58.590 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:18:58.660 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:18:58.660 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:18:59.377 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:18:59.490 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:00.070 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:00.357 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:00.357 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:00.357 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:00.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel-debuginfo 2023-12-06T09:19:01.128 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:07:52 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:01.274 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 108 k 2023-12-06T09:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 108 k 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 440 k 2023-12-06T09:19:01.276 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:01.904 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-debuginfo-18.0.0-7644.g96fa2578. 172 kB/s | 108 kB 00:00 2023-12-06T09:19:01.905 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:01.905 INFO:teuthology.orchestra.run.smithi132.stdout:Total 172 kB/s | 108 kB 00:00 2023-12-06T09:19:01.905 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:01.908 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:01.909 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:01.944 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:01.944 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:02.056 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:02.162 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:02.368 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:02.482 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:02.559 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:02.829 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:02.830 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:02.830 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:02.830 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:02.830 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:02.830 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:03.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2-debuginfo 2023-12-06T09:19:03.608 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:07:54 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:03.754 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.1 M 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:03.755 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:03.756 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:03.756 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:03.756 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:03.756 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 5.1 M 2023-12-06T09:19:03.756 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 24 M 2023-12-06T09:19:03.756 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:04.482 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs2-debuginfo-18.0.0-7644.g96fa2578.el9. 7.1 MB/s | 5.1 MB 00:00 2023-12-06T09:19:04.483 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:04.483 INFO:teuthology.orchestra.run.smithi132.stdout:Total 7.1 MB/s | 5.1 MB 00:00 2023-12-06T09:19:04.484 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:04.488 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:04.488 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:04.550 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:04.551 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:05.188 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:05.297 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:05.812 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:06.088 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:06.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:06.088 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:06.088 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:06.088 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:06.088 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:06.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2-debuginfo 2023-12-06T09:19:06.889 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:07:57 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:07.035 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 43 M 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 43 M 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 248 M 2023-12-06T09:19:07.036 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:09.277 INFO:teuthology.orchestra.run.smithi132.stdout:librados2-debuginfo-18.0.0-7644.g96fa2578.el9.x 19 MB/s | 43 MB 00:02 2023-12-06T09:19:09.277 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:09.278 INFO:teuthology.orchestra.run.smithi132.stdout:Total 19 MB/s | 43 MB 00:02 2023-12-06T09:19:09.278 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:09.283 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:09.283 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:09.579 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:09.579 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:10.741 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:11.050 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:11.050 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:11.051 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:11.051 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:11.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:11.051 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:11.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse-debuginfo 2023-12-06T09:19:11.833 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:56 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:11.979 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:11.980 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:11.981 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.8 M 2023-12-06T09:19:11.981 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:11.982 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:11.982 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 5.8 M 2023-12-06T09:19:11.982 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 28 M 2023-12-06T09:19:11.982 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:12.478 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x 12 MB/s | 5.8 MB 00:00 2023-12-06T09:19:12.479 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:12.479 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 5.8 MB 00:00 2023-12-06T09:19:12.479 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:12.482 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:12.482 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:12.549 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:12.549 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:13.265 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:13.367 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:13.941 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:14.222 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:14.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel-debuginfo 2023-12-06T09:19:14.513 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:14.608 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:15.005 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:00 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:15.151 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 108 k 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:15.152 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:15.153 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:15.153 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 108 k 2023-12-06T09:19:15.153 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 440 k 2023-12-06T09:19:15.153 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:15.480 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-debuginfo-18.0.0-7644.g96fa2578. 332 kB/s | 108 kB 00:00 2023-12-06T09:19:15.481 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:15.481 INFO:teuthology.orchestra.run.smithi088.stdout:Total 330 kB/s | 108 kB 00:00 2023-12-06T09:19:15.481 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:15.485 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:15.485 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:15.518 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:15.519 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:15.646 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:15.745 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:16.175 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:16.472 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:16.473 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:16.473 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:16.473 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:16.473 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:16.473 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:16.491 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:16.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2-debuginfo 2023-12-06T09:19:16.768 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:16.768 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:16.768 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:16.768 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:16.768 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:16.769 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:16.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1-debuginfo 2023-12-06T09:19:17.247 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:02 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:17.393 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:17.394 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:17.394 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:17.394 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.1 M 2023-12-06T09:19:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 5.1 M 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 24 M 2023-12-06T09:19:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:17.570 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:08 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:17.712 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:17.712 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:17.712 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:17.712 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 50 M 2023-12-06T09:19:17.713 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 294 M 2023-12-06T09:19:17.714 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:17.883 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs2-debuginfo-18.0.0-7644.g96fa2578.el9. 11 MB/s | 5.1 MB 00:00 2023-12-06T09:19:17.883 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:17.884 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 5.1 MB 00:00 2023-12-06T09:19:17.884 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:17.887 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:17.887 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:17.951 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:17.951 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:18.604 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:18.715 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:19.213 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:19.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:19.496 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:19.496 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:19.496 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:19.496 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:19.496 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:19.696 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2-debuginfo 2023-12-06T09:19:20.096 INFO:teuthology.orchestra.run.smithi132.stdout:librbd1-debuginfo-18.0.0-7644.g96fa2578.el9.x86 21 MB/s | 50 MB 00:02 2023-12-06T09:19:20.097 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:20.097 INFO:teuthology.orchestra.run.smithi132.stdout:Total 21 MB/s | 50 MB 00:02 2023-12-06T09:19:20.097 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:20.101 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:20.101 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:20.273 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:05 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:20.417 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 43 M 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:20.418 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:20.419 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 43 M 2023-12-06T09:19:20.419 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 248 M 2023-12-06T09:19:20.419 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:20.437 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:20.437 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:22.785 INFO:teuthology.orchestra.run.smithi088.stdout:librados2-debuginfo-18.0.0-7644.g96fa2578.el9.x 18 MB/s | 43 MB 00:02 2023-12-06T09:19:22.786 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:22.787 INFO:teuthology.orchestra.run.smithi088.stdout:Total 18 MB/s | 43 MB 00:02 2023-12-06T09:19:22.787 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:22.791 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:22.791 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:23.086 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:23.086 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:26.170 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:26.260 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:28.006 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:28.116 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:28.451 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:28.718 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:28.719 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:28.719 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:28.719 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:28.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs-debuginfo 2023-12-06T09:19:29.512 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:20 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:29.658 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 615 k 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 615 k 2023-12-06T09:19:29.659 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.1 M 2023-12-06T09:19:29.660 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:29.940 INFO:teuthology.orchestra.run.smithi132.stdout:python3-cephfs-debuginfo-18.0.0-7644.g96fa2578. 2.1 MB/s | 615 kB 00:00 2023-12-06T09:19:29.940 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:29.941 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.1 MB/s | 615 kB 00:00 2023-12-06T09:19:29.941 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:29.941 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:29.944 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:29.944 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:29.980 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:29.980 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:30.144 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T09:19:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:30.218 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:30.218 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:30.247 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:30.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1-debuginfo 2023-12-06T09:19:30.649 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:30.940 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:30.941 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:30.941 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:30.941 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:30.941 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:30.941 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:30.998 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:16 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:31.142 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:31.144 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 50 M 2023-12-06T09:19:31.144 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 294 M 2023-12-06T09:19:31.144 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:31.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados-debuginfo 2023-12-06T09:19:31.741 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:22 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:31.886 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:31.886 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 770 k 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 770 k 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.7 M 2023-12-06T09:19:31.887 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:32.977 INFO:teuthology.orchestra.run.smithi132.stdout:python3-rados-debuginfo-18.0.0-7644.g96fa2578.e 707 kB/s | 770 kB 00:01 2023-12-06T09:19:32.977 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:32.978 INFO:teuthology.orchestra.run.smithi132.stdout:Total 706 kB/s | 770 kB 00:01 2023-12-06T09:19:32.978 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:32.982 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:32.982 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:33.024 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:33.024 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:33.202 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:33.308 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T09:19:33.545 INFO:teuthology.orchestra.run.smithi088.stdout:librbd1-debuginfo-18.0.0-7644.g96fa2578.el9.x86 21 MB/s | 50 MB 00:02 2023-12-06T09:19:33.546 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:33.546 INFO:teuthology.orchestra.run.smithi088.stdout:Total 21 MB/s | 50 MB 00:02 2023-12-06T09:19:33.547 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:33.551 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:33.551 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:33.724 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T09:19:33.885 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:33.885 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:33.992 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T09:19:33.992 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:33.992 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:33.993 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:33.993 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:33.993 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:34.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd-debuginfo 2023-12-06T09:19:34.781 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:25 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:34.928 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:34.928 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.1 M 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.1 M 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 4.3 M 2023-12-06T09:19:34.929 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:35.396 INFO:teuthology.orchestra.run.smithi132.stdout:python3-rbd-debuginfo-18.0.0-7644.g96fa2578.el9 2.4 MB/s | 1.1 MB 00:00 2023-12-06T09:19:35.397 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:35.397 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2023-12-06T09:19:35.397 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:35.400 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:35.400 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:35.440 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:35.440 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:35.744 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:36.127 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:36.417 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:36.417 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:36.417 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:36.417 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:36.417 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:36.417 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:36.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw-debuginfo 2023-12-06T09:19:37.231 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:28 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:37.379 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:37.379 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:37.379 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:37.379 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:37.379 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:37.379 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 337 k 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 337 k 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.0 M 2023-12-06T09:19:37.380 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:37.766 INFO:teuthology.orchestra.run.smithi132.stdout:python3-rgw-debuginfo-18.0.0-7644.g96fa2578.el9 874 kB/s | 337 kB 00:00 2023-12-06T09:19:37.767 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:37.767 INFO:teuthology.orchestra.run.smithi132.stdout:Total 871 kB/s | 337 kB 00:00 2023-12-06T09:19:37.767 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:37.771 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:37.771 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:37.805 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:37.805 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:37.940 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:38.057 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:38.437 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:38.718 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:38.718 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:38.718 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:38.718 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:38.718 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:38.718 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:38.922 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse-debuginfo 2023-12-06T09:19:39.500 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:30 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:39.637 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:39.645 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 506 k 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:39.646 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:39.647 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:39.647 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 506 k 2023-12-06T09:19:39.647 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.5 M 2023-12-06T09:19:39.647 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:39.743 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:40.036 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x8 1.3 MB/s | 506 kB 00:00 2023-12-06T09:19:40.037 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:40.037 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.3 MB/s | 506 kB 00:00 2023-12-06T09:19:40.037 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:40.040 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:40.040 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:40.076 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:40.076 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:40.236 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:40.334 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:40.683 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:40.949 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:40.949 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:40.949 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:40.949 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:40.949 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:40.949 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:41.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror-debuginfo 2023-12-06T09:19:41.745 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:32 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:41.865 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:41.890 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:41.891 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:41.891 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:41.891 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:41.891 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:41.891 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 50 M 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 284 M 2023-12-06T09:19:41.892 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:42.142 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:42.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:42.142 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:42.142 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:42.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:42.142 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:42.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs-debuginfo 2023-12-06T09:19:42.902 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:27 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:43.049 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:43.050 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:43.050 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:43.050 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:43.050 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:43.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 615 k 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 615 k 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.1 M 2023-12-06T09:19:43.051 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:43.772 INFO:teuthology.orchestra.run.smithi088.stdout:python3-cephfs-debuginfo-18.0.0-7644.g96fa2578. 853 kB/s | 615 kB 00:00 2023-12-06T09:19:43.773 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:43.773 INFO:teuthology.orchestra.run.smithi088.stdout:Total 851 kB/s | 615 kB 00:00 2023-12-06T09:19:43.773 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:43.776 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:43.777 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:43.814 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:43.814 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:43.981 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:44.088 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:44.298 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-debuginfo-18.0.0-7644.g96fa2578.el9. 21 MB/s | 50 MB 00:02 2023-12-06T09:19:44.298 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:44.299 INFO:teuthology.orchestra.run.smithi132.stdout:Total 21 MB/s | 50 MB 00:02 2023-12-06T09:19:44.299 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:44.303 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:44.303 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:44.506 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:44.628 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:44.628 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:44.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T09:19:44.831 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:44.831 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:44.831 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:44.831 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:44.831 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:45.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados-debuginfo 2023-12-06T09:19:45.625 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:30 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:45.769 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:45.770 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:45.770 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:45.770 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:45.770 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 770 k 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 770 k 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.7 M 2023-12-06T09:19:45.771 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:46.209 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rados-debuginfo-18.0.0-7644.g96fa2578.e 1.7 MB/s | 770 kB 00:00 2023-12-06T09:19:46.209 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:46.209 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 770 kB 00:00 2023-12-06T09:19:46.210 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:46.213 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:46.213 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:46.252 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:46.252 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T09:19:46.958 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T09:19:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T09:19:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:47.259 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:47.259 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:47.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd-debuginfo 2023-12-06T09:19:48.048 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:33 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:48.194 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.1 M 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:48.195 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:48.196 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.1 M 2023-12-06T09:19:48.196 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 4.3 M 2023-12-06T09:19:48.196 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:48.495 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rbd-debuginfo-18.0.0-7644.g96fa2578.el9 3.8 MB/s | 1.1 MB 00:00 2023-12-06T09:19:48.496 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:48.496 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.7 MB/s | 1.1 MB 00:00 2023-12-06T09:19:48.496 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:48.499 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:48.499 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:48.539 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:48.539 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:48.758 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:48.852 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:49.231 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:49.523 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:49.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:49.523 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:49.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:49.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:49.523 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:49.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw-debuginfo 2023-12-06T09:19:50.306 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:50.311 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:35 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:50.403 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:50.456 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 337 k 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 337 k 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.0 M 2023-12-06T09:19:50.457 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:50.848 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rgw-debuginfo-18.0.0-7644.g96fa2578.el9 864 kB/s | 337 kB 00:00 2023-12-06T09:19:50.848 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:50.849 INFO:teuthology.orchestra.run.smithi088.stdout:Total 861 kB/s | 337 kB 00:00 2023-12-06T09:19:50.849 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:50.852 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:50.852 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:50.886 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:50.887 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:51.025 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:51.137 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:51.542 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:51.838 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T09:19:51.838 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:51.838 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:51.839 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:51.839 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:51.839 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:52.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse-debuginfo 2023-12-06T09:19:52.411 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:52.615 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:37 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:52.687 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:19:52.687 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:52.687 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:52.687 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:52.687 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:52.687 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:52.759 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:52.760 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:52.760 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:52.760 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:52.760 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:52.760 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 506 k 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 506 k 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.5 M 2023-12-06T09:19:52.761 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:52.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd-debuginfo 2023-12-06T09:19:53.143 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x8 1.3 MB/s | 506 kB 00:00 2023-12-06T09:19:53.143 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:53.144 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.3 MB/s | 506 kB 00:00 2023-12-06T09:19:53.144 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:53.146 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:53.146 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:53.182 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:53.182 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:19:53.338 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:19:53.432 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:08:44 ago on Wed 06 Dec 2023 09:11:09 AM UTC. 2023-12-06T09:19:53.651 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T09:19:53.652 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T09:19:53.652 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:53.652 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T09:19:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.2 M 2023-12-06T09:19:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.2 M 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 5.4 M 2023-12-06T09:19:53.653 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T09:19:53.831 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:54.122 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T09:19:54.122 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:54.122 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:19:54.123 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:54.123 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:54.123 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:19:54.124 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-debuginfo-18.0.0-7644.g96fa2578.el9.x86 2.6 MB/s | 1.2 MB 00:00 2023-12-06T09:19:54.125 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:54.125 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.6 MB/s | 1.2 MB 00:00 2023-12-06T09:19:54.125 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T09:19:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T09:19:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T09:19:54.170 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T09:19:54.171 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T09:19:54.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror-debuginfo 2023-12-06T09:19:54.401 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T09:19:54.515 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:54.894 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:39 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:19:54.949 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:55.041 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:19:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 50 M 2023-12-06T09:19:55.043 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 284 M 2023-12-06T09:19:55.043 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:19:55.223 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:19:55.223 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:55.223 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T09:19:55.223 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:19:55.223 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:19:55.223 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T09:19:55.447 DEBUG:teuthology.parallel:result is None 2023-12-06T09:19:57.720 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-debuginfo-18.0.0-7644.g96fa2578.el9. 19 MB/s | 50 MB 00:02 2023-12-06T09:19:57.721 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:19:57.722 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 50 MB 00:02 2023-12-06T09:19:57.722 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:19:57.726 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:19:57.726 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:19:58.050 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:19:58.050 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:20:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:20:03.826 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:20:05.802 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:20:06.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T09:20:06.112 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:06.112 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:20:06.112 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:20:06.112 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:06.112 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:20:06.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd-debuginfo 2023-12-06T09:20:06.889 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:51 ago on Wed 06 Dec 2023 09:11:15 AM UTC. 2023-12-06T09:20:07.033 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.2 M 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-12-06T09:20:07.034 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2023-12-06T09:20:07.035 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:07.035 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.2 M 2023-12-06T09:20:07.035 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 5.4 M 2023-12-06T09:20:07.035 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-12-06T09:20:07.496 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-debuginfo-18.0.0-7644.g96fa2578.el9.x86 2.7 MB/s | 1.2 MB 00:00 2023-12-06T09:20:07.496 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-12-06T09:20:07.497 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.7 MB/s | 1.2 MB 00:00 2023-12-06T09:20:07.497 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-12-06T09:20:07.500 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-12-06T09:20:07.501 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-12-06T09:20:07.542 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-12-06T09:20:07.542 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-12-06T09:20:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-12-06T09:20:07.894 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:20:08.331 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:20:08.628 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T09:20:08.628 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:08.628 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-12-06T09:20:08.628 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T09:20:08.629 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:08.629 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-12-06T09:20:08.813 DEBUG:teuthology.parallel:result is None 2023-12-06T09:20:08.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T09:20:08.963 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T09:20:08.984 INFO:teuthology.orchestra.run.smithi088.stdout:18.0.0-7644.g96fa2578.el9 2023-12-06T09:20:08.984 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7644.g96fa2578.el9 2023-12-06T09:20:08.985 INFO:teuthology.task.install:The correct ceph version 18.0.0-7644.g96fa2578 is installed. 2023-12-06T09:20:08.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T09:20:09.202 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T09:20:09.226 INFO:teuthology.orchestra.run.smithi132.stdout:18.0.0-7644.g96fa2578.el9 2023-12-06T09:20:09.226 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7644.g96fa2578.el9 2023-12-06T09:20:09.226 INFO:teuthology.task.install:The correct ceph version 18.0.0-7644.g96fa2578 is installed. 2023-12-06T09:20:09.226 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-06T09:20:09.227 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:09.227 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T09:20:09.254 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:09.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T09:20:09.298 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-06T09:20:09.299 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:09.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T09:20:09.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T09:20:09.392 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:09.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T09:20:09.422 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T09:20:09.491 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-06T09:20:09.491 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:09.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T09:20:09.518 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T09:20:09.585 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:09.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T09:20:09.613 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T09:20:09.681 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-06T09:20:09.681 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:09.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T09:20:09.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T09:20:09.775 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:09.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T09:20:09.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T09:20:09.873 INFO:teuthology.run_tasks:Running task ceph... 2023-12-06T09:20:09.949 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-06T09:20:09.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2023-12-06T09:20:09.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /var/log/ceph 2023-12-06T09:20:09.980 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-06T09:20:09.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-06T09:20:10.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-06T09:20:10.046 INFO:tasks.ceph:Creating extra log directories... 2023-12-06T09:20:10.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-06T09:20:10.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-06T09:20:10.125 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-06T09:20:10.125 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-06T09:20:10.126 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622', 'branch': 'wip-sseshasa-testing-2023-12-04-1848', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7479622', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-sseshasa-testing-2023-12-04-1848'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-sseshasa-testing-2023-12-04-1848', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}}, 'owner': 'scheduled_sseshasa@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5341, 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'sleep_before_teardown': 0, 'subset': '222/150000', 'suite': 'rados', 'suite_branch': 'wip-sseshasa-testing-2023-12-04-1848', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHHTCpfjfHqstmkXRAr29MBgjiE9LKJ7mmVdbETF57yMeJIhqBvVHbaKtJ/N+dGuo6+PDjx/O8tQ0aC31Q0dv0=', 'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwTJzvIaMcfYCeVIdChPPodcm3xbZK0f0XHf0RcDpgGKOBNDbbi3jNe6hMMeG4nf8PxhSd4BZBMjCWL9Zi/qqA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'mon_recovery': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-06_06:16:39', 'tube': 'smithi', 'user': 'sseshasa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-06T09:20:10.126 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-06T09:20:10.153 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-06T09:20:10.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T09:20:10.208 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T09:20:10.235 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:10.235 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-12-06T09:20:10.288 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-06T09:20:10.289 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-12-06T09:20:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-06T09:20:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:10.347 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-06T09:20:10.347 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:10.348 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:10.348 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-12-06 09:12:59.008305253 +0000 2023-12-06T09:20:10.348 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-12-06 09:10:14.422583307 +0000 2023-12-06T09:20:10.348 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-12-06 09:10:14.422583307 +0000 2023-12-06T09:20:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-12-06 09:10:14.422583307 +0000 2023-12-06T09:20:10.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-06T09:20:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-12-06T09:20:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-12-06T09:20:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000203246 s, 2.5 MB/s 2023-12-06T09:20:10.414 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-06T09:20:10.470 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-12-06 09:12:59.008305253 +0000 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-12-06 09:10:14.918572880 +0000 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-12-06 09:10:14.918572880 +0000 2023-12-06T09:20:10.526 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-12-06 09:10:14.918572880 +0000 2023-12-06T09:20:10.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-06T09:20:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-12-06T09:20:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-12-06T09:20:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000209506 s, 2.4 MB/s 2023-12-06T09:20:10.592 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-06T09:20:10.648 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-12-06T09:20:10.702 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-12-06 09:12:59.009305233 +0000 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-12-06 09:10:15.420562327 +0000 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-12-06 09:10:15.420562327 +0000 2023-12-06T09:20:10.703 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-12-06 09:10:15.420562327 +0000 2023-12-06T09:20:10.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-06T09:20:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-12-06T09:20:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-12-06T09:20:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000185924 s, 2.8 MB/s 2023-12-06T09:20:10.769 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-06T09:20:10.825 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-12-06T09:20:10.879 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-06T09:20:10.879 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:10.879 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-06T09:20:10.879 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:10.879 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:10.879 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-12-06 09:12:59.009305233 +0000 2023-12-06T09:20:10.880 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-12-06 09:10:15.899552257 +0000 2023-12-06T09:20:10.880 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-12-06 09:10:15.899552257 +0000 2023-12-06T09:20:10.880 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-12-06 09:10:15.899552257 +0000 2023-12-06T09:20:10.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-06T09:20:10.944 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-12-06T09:20:10.944 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-12-06T09:20:10.945 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000176967 s, 2.9 MB/s 2023-12-06T09:20:10.946 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-06T09:20:11.001 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:11.002 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:11.002 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2023-12-06T09:20:11.020 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-06T09:20:11.020 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2023-12-06 09:12:48.677960483 +0000 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2023-12-06 09:10:14.421647584 +0000 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2023-12-06 09:10:14.421647584 +0000 2023-12-06T09:20:11.078 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2023-12-06 09:10:14.421647584 +0000 2023-12-06T09:20:11.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-06T09:20:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2023-12-06T09:20:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2023-12-06T09:20:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000217943 s, 2.3 MB/s 2023-12-06T09:20:11.149 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-06T09:20:11.207 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2023-12-06 09:12:48.678960457 +0000 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2023-12-06 09:10:14.920634588 +0000 2023-12-06T09:20:11.264 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2023-12-06 09:10:14.920634588 +0000 2023-12-06T09:20:11.265 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2023-12-06 09:10:14.920634588 +0000 2023-12-06T09:20:11.265 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-06T09:20:11.332 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2023-12-06T09:20:11.333 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2023-12-06T09:20:11.333 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000275156 s, 1.9 MB/s 2023-12-06T09:20:11.335 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-06T09:20:11.392 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2023-12-06T09:20:11.447 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-06T09:20:11.447 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:11.447 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-06T09:20:11.447 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:11.448 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:11.448 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2023-12-06 09:12:48.678960457 +0000 2023-12-06T09:20:11.448 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2023-12-06 09:10:15.426621409 +0000 2023-12-06T09:20:11.448 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2023-12-06 09:10:15.426621409 +0000 2023-12-06T09:20:11.448 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2023-12-06 09:10:15.426621409 +0000 2023-12-06T09:20:11.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-06T09:20:11.515 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2023-12-06T09:20:11.515 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2023-12-06T09:20:11.515 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.00026875 s, 1.9 MB/s 2023-12-06T09:20:11.516 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-06T09:20:11.573 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2023-12-06T09:20:11.628 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2023-12-06 09:12:48.678960457 +0000 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2023-12-06 09:10:15.938608074 +0000 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2023-12-06 09:10:15.938608074 +0000 2023-12-06T09:20:11.629 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2023-12-06 09:10:15.938608074 +0000 2023-12-06T09:20:11.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-06T09:20:11.694 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2023-12-06T09:20:11.694 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2023-12-06T09:20:11.695 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000176564 s, 2.9 MB/s 2023-12-06T09:20:11.695 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-06T09:20:11.751 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:11.751 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-06T09:20:11.751 INFO:tasks.ceph:Generating config... 2023-12-06T09:20:11.752 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-06T09:20:11.753 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-12-06T09:20:11.753 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-12-06T09:20:11.753 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-12-06T09:20:11.753 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-06T09:20:11.753 INFO:tasks.ceph:[global] ms type = async 2023-12-06T09:20:11.753 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-06T09:20:11.753 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-06T09:20:11.754 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-06T09:20:11.754 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-06T09:20:11.754 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-06T09:20:11.755 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-06T09:20:11.755 INFO:tasks.ceph:Setting up mon.a... 2023-12-06T09:20:11.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-06T09:20:11.807 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2023-12-06T09:20:11.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-06T09:20:11.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-06T09:20:11.883 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.88'), ('mon.c', '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]'), ('mon.b', '172.21.15.132')] 2023-12-06T09:20:11.884 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],172.21.15.132', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-06T09:20:11.885 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:11.885 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T09:20:11.941 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.88 --addv c '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]' --add b 172.21.15.132 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:12.025 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid 0c48cca3-dcc0-44bd-8eed-37fea2138550 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = pacific 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:fsid 0c48cca3-dcc0-44bd-8eed-37fea2138550 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2023-12-06T09:20:12.041363+0000 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-12-06T09:20:12.041363+0000 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 16 (pacific) 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 3 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.b 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.c 2023-12-06T09:20:12.026 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-06T09:20:12.028 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T09:20:12.042 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0c48cca3-dcc0-44bd-8eed-37fea2138550... 2023-12-06T09:20:12.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-06T09:20:12.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-06T09:20:12.132 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2023-12-06T09:20:12.132 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2023-12-06T09:20:12.132 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T09:20:12.132 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.133 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2023-12-06T09:20:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],172.21.15.132" 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: mon election default strategy = 3 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: ms bind msgr1 = False 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: ms bind msgr2 = True 2023-12-06T09:20:12.135 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 5000 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: ms type = async 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd heartbeat grace = 80 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 0c48cca3-dcc0-44bd-8eed-37fea2138550 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2023-12-06T09:20:12.136 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug trim objects = true 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2023-12-06T09:20:12.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore compression algorithm = zstd 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore compression mode = aggressive 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore zero block detection = True 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 1 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 1 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2023-12-06T09:20:12.138 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd blocked scrub grace period = 3600 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify cached snaps = True 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify missing on start = True 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd fast shutdown = False 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd max markdown count = 1000 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock override recovery settings = True 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue = debug_random 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue cut off = debug_random 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: osd shutdown pgref assert = True 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-12-06T09:20:12.139 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2023-12-06T09:20:12.140 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon min osdmap epochs = 50 2023-12-06T09:20:12.141 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd crush smoke test = False 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune interval = 2 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune min = 15 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune txsize = 2 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub interval = 300 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: paxos service trim min = 10 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2023-12-06T09:20:12.142 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.143 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2023-12-06T09:20:12.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.145 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2023-12-06T09:20:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2023-12-06T09:20:12.146 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = "172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],172.21.15.132" 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: mon client directed command retry = 5 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: mon election default strategy = 3 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: ms bind msgr1 = False 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: ms bind msgr2 = True 2023-12-06T09:20:12.147 INFO:teuthology.orchestra.run.smithi132.stdout: ms inject socket failures = 5000 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: ms type = async 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: osd heartbeat grace = 80 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = 0c48cca3-dcc0-44bd-8eed-37fea2138550 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.148 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.149 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug trim objects = true 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2023-12-06T09:20:12.150 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.151 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore block size = 96636764160 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore compression algorithm = zstd 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore compression mode = aggressive 2023-12-06T09:20:12.152 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore fsck on mount = True 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore zero block detection = True 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluefs = 1 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluestore = 1 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: debug monc = 20 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: debug rocksdb = 10 2023-12-06T09:20:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd full ratio = 0.9 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd nearfull ratio = 0.8 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd blocked scrub grace period = 3600 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify cached snaps = True 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify missing on start = True 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd failsafe full ratio = 0.95 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd fast shutdown = False 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd max markdown count = 1000 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock override recovery settings = True 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd objectstore = bluestore 2023-12-06T09:20:12.154 INFO:teuthology.orchestra.run.smithi132.stdout: osd op queue = debug_random 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: osd op queue cut off = debug_random 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: osd shutdown pgref assert = True 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2023-12-06T09:20:12.155 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T09:20:12.156 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon min osdmap epochs = 50 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd crush smoke test = False 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon osdmap full prune interval = 2 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon osdmap full prune min = 15 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon osdmap full prune txsize = 2 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: mon scrub interval = 300 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: paxos service trim min = 10 2023-12-06T09:20:12.157 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T09:20:12.158 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2023-12-06T09:20:12.159 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.c] 2023-12-06T09:20:12.159 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.b] 2023-12-06T09:20:12.160 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-06T09:20:12.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-06T09:20:12.255 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-06T09:20:12.255 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:12.255 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-06T09:20:12.270 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:12.270 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-06T09:20:12.325 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:20:12.325 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:12.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-06T09:20:12.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-06T09:20:12.406 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:12.406 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:12.461 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:20:12.461 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:12.461 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-06T09:20:12.461 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-06T09:20:12.501 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:12.501 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:12.556 INFO:tasks.ceph:Setting up mon nodes... 2023-12-06T09:20:12.556 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-06T09:20:12.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-06T09:20:12.612 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-06T09:20:12.615 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-06T09:20:12.682 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-06T09:20:12.683 INFO:tasks.ceph:Setting up mds nodes... 2023-12-06T09:20:12.684 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-06T09:20:12.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-06T09:20:12.728 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-06T09:20:12.745 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-06T09:20:12.796 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-06T09:20:12.811 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-06T09:20:12.812 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-06T09:20:12.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-06T09:20:12.845 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:12.846 INFO:tasks.ceph:role: osd.0 2023-12-06T09:20:12.846 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:20:12.846 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-06T09:20:12.922 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:12.922 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:12.923 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:13.416 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-12-06T09:20:13.418 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-12-06T09:20:13.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-06T09:20:13.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-06T09:20:13.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-06T09:20:13.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:13.634 INFO:tasks.ceph:role: osd.1 2023-12-06T09:20:13.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:20:13.634 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-06T09:20:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:14.170 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-12-06T09:20:14.172 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-12-06T09:20:14.172 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-06T09:20:14.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-06T09:20:14.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-06T09:20:14.374 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:14.375 INFO:tasks.ceph:role: osd.2 2023-12-06T09:20:14.375 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:20:14.375 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-06T09:20:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:14.448 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:14.448 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:14.448 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:14.449 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:14.449 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:14.923 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-12-06T09:20:14.925 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-12-06T09:20:14.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-06T09:20:15.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-06T09:20:15.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-06T09:20:15.175 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:15.176 INFO:tasks.ceph:role: osd.3 2023-12-06T09:20:15.176 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:20:15.176 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-06T09:20:15.249 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:15.250 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:15.782 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-12-06T09:20:15.784 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-12-06T09:20:15.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-06T09:20:15.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-06T09:20:15.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:16.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:16.053+0000 7fb99d08c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-06T09:20:16.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:16.054+0000 7fb99d08c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-06T09:20:16.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:16.054+0000 7fb99d08c500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-06T09:20:16.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:16.054+0000 7fb99d08c500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-06T09:20:16.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:16.054+0000 7fb99d08c500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-06T09:20:16.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:16.054+0000 7fb99d08c500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-06T09:20:18.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-06T09:20:18.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:19.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:19.023+0000 7f92162ab500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-06T09:20:19.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:19.023+0000 7f92162ab500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-06T09:20:19.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:19.023+0000 7f92162ab500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-06T09:20:19.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:19.023+0000 7f92162ab500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-06T09:20:19.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:19.023+0000 7f92162ab500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-06T09:20:19.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:19.024+0000 7f92162ab500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-06T09:20:21.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-06T09:20:21.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:21.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:21.982+0000 7f1c1e994500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-06T09:20:21.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:21.982+0000 7f1c1e994500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-06T09:20:21.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:21.982+0000 7f1c1e994500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-06T09:20:21.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:21.982+0000 7f1c1e994500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-06T09:20:21.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:21.982+0000 7f1c1e994500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-06T09:20:21.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:21.982+0000 7f1c1e994500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-06T09:20:24.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-06T09:20:24.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:24.944+0000 7f502780e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-06T09:20:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:24.944+0000 7f502780e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-06T09:20:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:24.944+0000 7f502780e500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-06T09:20:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:24.944+0000 7f502780e500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-06T09:20:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:24.944+0000 7f502780e500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-06T09:20:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:20:24.944+0000 7f502780e500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-06T09:20:27.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-06T09:20:27.801 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-06T09:20:27.833 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:27.833 INFO:tasks.ceph:role: osd.4 2023-12-06T09:20:27.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:20:27.833 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-06T09:20:27.911 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:27.911 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:27.911 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:27.912 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:28.419 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2023-12-06T09:20:28.421 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2023-12-06T09:20:28.421 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-06T09:20:28.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-06T09:20:28.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-06T09:20:28.647 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:28.647 INFO:tasks.ceph:role: osd.5 2023-12-06T09:20:28.647 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:20:28.647 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-06T09:20:28.724 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:28.724 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:28.724 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:28.725 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:28.725 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:28.725 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:28.725 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:29.182 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2023-12-06T09:20:29.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2023-12-06T09:20:29.184 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-06T09:20:29.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-06T09:20:29.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-06T09:20:29.410 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:29.410 INFO:tasks.ceph:role: osd.6 2023-12-06T09:20:29.410 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:20:29.410 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:29.484 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:30.006 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2023-12-06T09:20:30.008 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2023-12-06T09:20:30.008 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-06T09:20:30.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-06T09:20:30.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-06T09:20:30.255 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T09:20:30.255 INFO:tasks.ceph:role: osd.7 2023-12-06T09:20:30.255 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:20:30.255 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-06T09:20:30.330 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T09:20:30.330 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T09:20:30.330 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T09:20:30.331 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T09:20:30.840 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2023-12-06T09:20:30.846 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2023-12-06T09:20:30.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-06T09:20:30.929 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-06T09:20:30.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:31.063 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:31.077+0000 7f399fdbc500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-06T09:20:31.063 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:31.077+0000 7f399fdbc500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-06T09:20:31.063 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:31.077+0000 7f399fdbc500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-06T09:20:31.063 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:31.077+0000 7f399fdbc500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-06T09:20:31.063 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:31.077+0000 7f399fdbc500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-06T09:20:31.064 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:31.078+0000 7f399fdbc500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-06T09:20:33.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-06T09:20:33.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:34.024 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:34.038+0000 7fc1a3225500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-06T09:20:34.024 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:34.038+0000 7fc1a3225500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-06T09:20:34.024 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:34.038+0000 7fc1a3225500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-06T09:20:34.024 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:34.038+0000 7fc1a3225500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-06T09:20:34.024 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:34.038+0000 7fc1a3225500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-06T09:20:34.024 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:34.039+0000 7fc1a3225500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-06T09:20:36.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-06T09:20:36.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:36.990 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:37.002+0000 7fd292412500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-06T09:20:36.991 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:37.003+0000 7fd292412500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-06T09:20:36.991 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:37.003+0000 7fd292412500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-06T09:20:36.991 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:37.003+0000 7fd292412500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-06T09:20:36.991 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:37.003+0000 7fd292412500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-06T09:20:36.991 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:37.003+0000 7fd292412500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-06T09:20:39.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-06T09:20:39.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:39.972+0000 7fb326010500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-06T09:20:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:39.972+0000 7fb326010500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-06T09:20:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:39.972+0000 7fb326010500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-06T09:20:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:39.972+0000 7fb326010500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-06T09:20:39.959 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:39.972+0000 7fb326010500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-06T09:20:39.959 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T09:20:39.973+0000 7fb326010500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-06T09:20:42.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-06T09:20:42.827 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-06T09:20:42.827 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:42.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-06T09:20:42.859 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:42.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-06T09:20:42.931 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:42.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-06T09:20:43.002 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:43.002 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-06T09:20:43.072 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:43.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-06T09:20:43.144 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:43.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-06T09:20:43.175 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:43.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-06T09:20:43.247 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:43.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-06T09:20:43.316 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:43.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-06T09:20:43.384 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:43.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-06T09:20:43.452 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:20:43.452 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-06T09:20:43.470 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:20:43.470 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-06T09:20:43.507 INFO:tasks.ceph:Adding keys to all mons... 2023-12-06T09:20:43.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-06T09:20:43.513 DEBUG:teuthology.orchestra.run.smithi132:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-06T09:20:43.550 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDMPHBlSnF1JRAAWz7ObWtdVB9BRAmzmWyogw== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDQPHBlPuFGAxAAr2yZOs2UO4KEJBAETl/tug== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDTPHBlVox2ARAAM4ISEbOjzjh3t48DKxRSXw== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDVPHBl6JKXOhAAgNmFUgaKWmm9DNMd+JS7Iw== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDYPHBlvb5UOBAA/pm19NtYFYzRZLMcusMfig== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDMPHBlGqOJKRAAEkfz2Vp4tAvVHVRODfPoyA== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDfPHBl7pmtBBAATIbts8vWSoCF7uhCZZhw0A== 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2023-12-06T09:20:43.551 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDiPHBlBy9aAhAAihlAjdgs65YH3pV2SzA8Lw== 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDlPHBlox87ABAAJw6jB4J41hDeVIF5mBuYMA== 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDnPHBl0yAHOhAA12ZkRGuHDAt3ATmpWQBjYg== 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDMPHBlCV9RLBAAYij98YHBebNPVJ/06h2EoQ== 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2023-12-06T09:20:43.552 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDMPHBll7hYMBAAA7avtcZcksYwUJmP6CmaHg== 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.y] 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDMPHBlSnF1JRAAWz7ObWtdVB9BRAmzmWyogw== 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDQPHBlPuFGAxAAr2yZOs2UO4KEJBAETl/tug== 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDTPHBlVox2ARAAM4ISEbOjzjh3t48DKxRSXw== 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2023-12-06T09:20:43.575 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDVPHBl6JKXOhAAgNmFUgaKWmm9DNMd+JS7Iw== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.3] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDYPHBlvb5UOBAA/pm19NtYFYzRZLMcusMfig== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.x] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDMPHBlGqOJKRAAEkfz2Vp4tAvVHVRODfPoyA== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.4] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDfPHBl7pmtBBAATIbts8vWSoCF7uhCZZhw0A== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.5] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDiPHBlBy9aAhAAihlAjdgs65YH3pV2SzA8Lw== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.6] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDlPHBlox87ABAAJw6jB4J41hDeVIF5mBuYMA== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.7] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDnPHBl0yAHOhAA12ZkRGuHDAt3ATmpWQBjYg== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDMPHBlCV9RLBAAYij98YHBebNPVJ/06h2EoQ== 2023-12-06T09:20:43.576 INFO:teuthology.orchestra.run.smithi132.stdout:[client.1] 2023-12-06T09:20:43.577 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDMPHBll7hYMBAAA7avtcZcksYwUJmP6CmaHg== 2023-12-06T09:20:43.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T09:20:43.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T09:20:43.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.885 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T09:20:43.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T09:20:43.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.940 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:43.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:44.046 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:44.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:44.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:44.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T09:20:44.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T09:20:44.155 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T09:20:44.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T09:20:44.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T09:20:44.267 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-06T09:20:44.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-06T09:20:44.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-06T09:20:44.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-06T09:20:44.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-06T09:20:44.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-06T09:20:44.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-06T09:20:44.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-06T09:20:44.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-06T09:20:44.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-06T09:20:44.764 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:44.766 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-06T09:20:44.819 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-06T09:20:44.820 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-06T09:20:44.820 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T09:20:44.820 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T09:20:44.822 INFO:tasks.ceph.mon.a:Started 2023-12-06T09:20:44.822 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-06T09:20:44.822 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T09:20:44.823 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T09:20:44.824 INFO:tasks.ceph.mon.c:Started 2023-12-06T09:20:44.824 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-06T09:20:44.825 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T09:20:44.825 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T09:20:44.862 INFO:tasks.ceph.mon.b:Started 2023-12-06T09:20:44.862 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-06T09:20:44.862 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-06T09:20:44.862 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-06T09:20:44.869 INFO:tasks.ceph.mgr.y:Started 2023-12-06T09:20:44.869 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-06T09:20:44.869 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-06T09:20:44.872 INFO:tasks.ceph.mgr.x:Started 2023-12-06T09:20:44.872 INFO:tasks.ceph:Setting crush tunables to default 2023-12-06T09:20:44.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2023-12-06T09:20:52.502 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:07.535 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:20:52.687 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:07.760 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:20:52.690 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:07.764 86645== Warning: unimplemented fcntl command: 1036 2023-12-06T09:20:52.816 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:07.850 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:20:53.014 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.086 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:20:53.017 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.091 86645== Warning: unimplemented fcntl command: 1036 2023-12-06T09:21:09.788 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2023-12-06T09:21:09.802 INFO:tasks.ceph:check_enable_crimson: False 2023-12-06T09:21:09.802 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-06T09:21:09.803 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:21:09.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-06T09:21:09.831 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:21:09.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-06T09:21:09.854 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:09.868+0000 7f09f5c14040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-06T09:21:09.865 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:09.880+0000 7f1e0d832040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-06T09:21:09.882 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:21:09.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-06T09:21:09.908 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:09.922+0000 7f09f5c14040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-06T09:21:09.916 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:09.930+0000 7f1e0d832040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-06T09:21:09.935 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-12-06T09:21:09.935 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-06T09:21:10.003 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:21:10.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-06T09:21:10.033 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:21:10.033 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-06T09:21:10.103 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:21:10.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-06T09:21:10.170 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T09:21:10.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-06T09:21:10.237 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new db81b709-a318-4b43-beb2-bf3687f365e1 0 2023-12-06T09:21:10.488 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.502+0000 7f1e0d832040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-06T09:21:10.512 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.526+0000 7f09f5c14040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-06T09:21:10.552 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.567+0000 7f1e0d832040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-06T09:21:10.567 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.579+0000 7f09f5c14040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-06T09:21:10.608 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.623+0000 7f09f5c14040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-06T09:21:10.612 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.626+0000 7f1e0d832040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-06T09:21:10.661 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.676+0000 7f09f5c14040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-06T09:21:10.665 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.679+0000 7f1e0d832040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-06T09:21:10.707 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.721+0000 7f09f5c14040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-06T09:21:10.708 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.722+0000 7f1e0d832040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-06T09:21:10.754 INFO:teuthology.orchestra.run.smithi132.stdout:0 2023-12-06T09:21:10.770 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new fd7b0482-e237-424b-8eff-8db905713049 1 2023-12-06T09:21:10.814 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.827+0000 7f1e0d832040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-06T09:21:10.828 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.843+0000 7f09f5c14040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-06T09:21:10.888 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.903+0000 7f1e0d832040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-06T09:21:10.904 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.918+0000 7f09f5c14040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-06T09:21:10.936 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:10.950+0000 7f1e0d832040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-06T09:21:10.951 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:10.965+0000 7f09f5c14040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-06T09:21:10.987 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.001+0000 7f1e0d832040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-06T09:21:11.005 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.019+0000 7f09f5c14040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-06T09:21:11.036 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.051+0000 7f1e0d832040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-06T09:21:11.065 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.079+0000 7f09f5c14040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-06T09:21:11.139 INFO:teuthology.orchestra.run.smithi132.stdout:1 2023-12-06T09:21:11.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new e5a571be-6b8e-4c6f-bd3e-2af30c112285 2 2023-12-06T09:21:11.431 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.445+0000 7f1e0d832040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-06T09:21:11.464 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.479+0000 7f09f5c14040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-06T09:21:11.560 INFO:teuthology.orchestra.run.smithi132.stdout:2 2023-12-06T09:21:11.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new d7abe19a-cc3d-411f-8533-8592e1379735 3 2023-12-06T09:21:11.573 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.585+0000 7f1e0d832040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-06T09:21:11.606 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.619+0000 7f09f5c14040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-06T09:21:11.694 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.709+0000 7f1e0d832040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-06T09:21:11.718 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.732+0000 7f09f5c14040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-06T09:21:11.740 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.754+0000 7f1e0d832040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-06T09:21:11.765 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.779+0000 7f09f5c14040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-06T09:21:11.864 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.879+0000 7f09f5c14040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-06T09:21:11.874 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:11.888+0000 7f1e0d832040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-06T09:21:11.960 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:11.974+0000 7f09f5c14040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-06T09:21:11.997 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:12.011+0000 7f1e0d832040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-06T09:21:12.005 INFO:teuthology.orchestra.run.smithi132.stdout:3 2023-12-06T09:21:12.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new b9517160-d081-44f9-aea4-6d2529e02114 4 2023-12-06T09:21:12.114 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:12.126+0000 7f09f5c14040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-06T09:21:12.160 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:12.171+0000 7f1e0d832040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-06T09:21:12.161 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:12.176+0000 7f09f5c14040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-06T09:21:12.198 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:12.212+0000 7f1e0d832040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-06T09:21:12.306 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:12.321+0000 7f09f5c14040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-06T09:21:12.335 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:12.349+0000 7f1e0d832040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-06T09:21:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:4 2023-12-06T09:21:12.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e 5 2023-12-06T09:21:12.895 INFO:teuthology.orchestra.run.smithi132.stdout:5 2023-12-06T09:21:12.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new e7afeca8-1363-4e9b-96f6-7fab6113d9e0 6 2023-12-06T09:21:13.138 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:13.152+0000 7f09f5c14040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-06T09:21:13.151 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:13.164+0000 7f1e0d832040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-06T09:21:13.263 INFO:teuthology.orchestra.run.smithi132.stdout:6 2023-12-06T09:21:13.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd new fd4b3971-3adb-44fd-b42b-b6660305beb3 7 2023-12-06T09:21:13.428 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:13.442+0000 7f09f5c14040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-06T09:21:13.432 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:13.446+0000 7f1e0d832040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-06T09:21:13.479 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:13.493+0000 7f09f5c14040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-06T09:21:13.484 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:13.498+0000 7f1e0d832040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-06T09:21:13.580 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:13.594+0000 7f09f5c14040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-06T09:21:13.590 INFO:tasks.ceph.mgr.x.smithi132.stderr:2023-12-06T09:21:13.604+0000 7f1e0d832040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-06T09:21:13.725 INFO:teuthology.orchestra.run.smithi132.stdout:7 2023-12-06T09:21:13.734 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.735 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-06T09:21:13.735 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-06T09:21:13.737 INFO:tasks.ceph.osd.0:Started 2023-12-06T09:21:13.738 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.738 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-06T09:21:13.738 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-06T09:21:13.740 INFO:tasks.ceph.osd.1:Started 2023-12-06T09:21:13.740 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.740 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-06T09:21:13.741 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-06T09:21:13.743 INFO:tasks.ceph.osd.2:Started 2023-12-06T09:21:13.743 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.743 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-06T09:21:13.743 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-06T09:21:13.745 INFO:tasks.ceph.osd.3:Started 2023-12-06T09:21:13.745 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.745 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-06T09:21:13.745 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-06T09:21:13.747 INFO:tasks.ceph.osd.4:Started 2023-12-06T09:21:13.747 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.748 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-06T09:21:13.748 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-06T09:21:13.749 INFO:tasks.ceph.osd.5:Started 2023-12-06T09:21:13.749 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.750 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-06T09:21:13.750 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-06T09:21:13.752 INFO:tasks.ceph.osd.6:Started 2023-12-06T09:21:13.752 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T09:21:13.752 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-06T09:21:13.752 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-06T09:21:13.764 INFO:tasks.ceph.osd.7:Started 2023-12-06T09:21:13.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:14.667 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:21:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":10,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:13.597313+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:14.683 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-06T09:21:14.683 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-06T09:21:15.435 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:15.450+0000 7f0996b27640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-06T09:21:15.489 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:15.492 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:21:15.506+0000 7f0996b27640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-06T09:21:16.353 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:21:16.353 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:13.597313+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:16.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T09:21:21.771 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:07.960 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:21.833 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:08.022 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:21.834 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:08.023 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:21.867 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:08.052 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:21.892 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:08.082 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:21.929 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:08.114 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:21.972 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:08.145 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.030 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:08.203 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.285 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:08.478 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.290 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:08.464 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.348 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:08.522 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.348 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:08.540 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.380 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:21:22.388+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:22.443 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:21:22.448+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:22.447 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:08.636 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.457 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:08.628 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.473 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:21:22.481+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:22.506 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:08.695 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.520 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:08.691 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.622 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:21:22.631+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:22.815 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:09.004 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.901 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:09.091 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:22.904 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:21:22.910+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:22.909 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:21:22.918+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:22.930 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:09.117 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.085 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:09.257 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.092 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:21:23.101+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:23.099 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:21:23.106+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:21:23.177 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:23.198 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:09.387 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.290 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:09.479 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.317 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:09.503 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.373 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:09.547 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.404 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:09.595 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.466 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:09.638 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.488 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:09.678 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.549 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:09.720 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.555 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:09.745 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.588 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:09.774 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.615 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:09.801 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.763 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:09.937 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.764 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:09.953 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.767 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:09.940 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.784 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:09.976 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.793 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:21:23.793 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:13.597313+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:23.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T09:21:23.851 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:10.040 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.898 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:10.084 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.949 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:10.121 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:23.973 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.160 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.053 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:10.243 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.063 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:10.235 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.094 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.268 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.124 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:10.316 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.128 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:10.317 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.178 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:10.364 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.289 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:10.460 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.319 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.508 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.348 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:10.537 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.358 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:10.531 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.391 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.566 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.416 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:10.605 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.418 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:10.611 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.465 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:10.651 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.595 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:10.767 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.621 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.810 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.635 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:10.825 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.651 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:10.825 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.679 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.854 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.704 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:10.893 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.713 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:10.903 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.753 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:10.939 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.884 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:11.055 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.911 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:11.100 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.923 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:11.113 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.944 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:11.117 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.966 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:11.141 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:24.993 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:11.182 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.003 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:11.195 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.041 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:11.227 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.174 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:11.345 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.206 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:11.395 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.211 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:11.400 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.238 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:11.410 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.259 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:11.434 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.286 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:11.475 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.291 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:11.484 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.327 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:11.513 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.461 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:11.633 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.503 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:11.692 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.533 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:11.703 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.549 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:11.721 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.588 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:11.781 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.748 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:11.937 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.753 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:11.924 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.779 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:11.968 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.793 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:11.982 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.835 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:12.024 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.846 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:12.021 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.866 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:12.055 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.876 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:12.069 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.910 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:12.096 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:25.952 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:12.138 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.039 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:12.211 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.083 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:12.272 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.085 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:12.258 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.116 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:12.289 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.391 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:12.566 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.420 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:12.612 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.437 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:12.610 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.461 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:12.652 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.585 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:12.755 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.636 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:12.807 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.656 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:12.844 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:26.694 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:12.882 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:30.615 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:31.246 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:17.435 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:31.308 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:21:31.308 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:13.597313+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:31.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T09:21:31.695 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:17.884 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:31.888 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:18.074 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:31.955 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:18.128 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:32.053 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:18.224 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:32.182 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:18.372 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:32.588 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:18.777 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:32.650 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:18.821 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:36.872 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:23.061 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:36.878 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:23.065 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.340 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:23.529 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.343 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:23.533 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.384 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:23.558 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.388 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:23.562 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.722 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:23.895 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.729 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:23.902 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.778 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:23.964 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:37.784 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:23.970 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.130 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:38.143 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:24.332 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.147 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:24.336 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.396 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:24.588 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.400 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:24.592 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.490 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:24.661 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.493 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:24.665 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:38.821 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:21:38.821 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:13.597313+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:38.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T09:21:40.798 INFO:tasks.ceph.osd.4.smithi132.stderr:==00:00:00:26.988 86681== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:41.334 INFO:tasks.ceph.osd.5.smithi132.stderr:==00:00:00:27.523 86682== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:41.638 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:27.812 86971== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:41.920 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:28.093 86684== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:42.117 INFO:tasks.ceph.osd.6.smithi132.stderr:==00:00:00:28.303 86683== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:42.502 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:28.691 86969== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:42.861 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:29.032 86972== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:43.156 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:29.348 86966== Warning: unimplemented fcntl command: 1038 2023-12-06T09:21:45.643 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:46.254 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:21:46.254 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:13.597313+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:46.270 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T09:21:48.472 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:21:48.483+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-06T09:21:49.337 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:21:49.346+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-06T09:21:49.791 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:21:49.803+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-06T09:21:49.919 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:21:49.922+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-06T09:21:50.097 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:21:50.062+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-06T09:21:50.920 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:21:50.932+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-06T09:21:51.256 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:21:51.268+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-06T09:21:51.297 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:21:51.307+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-06T09:21:52.147 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:21:52.153+0000 bc92640 -1 osd.4 0 waiting for initial osdmap 2023-12-06T09:21:52.201 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:21:52.211+0000 b47e640 -1 osd.5 0 waiting for initial osdmap 2023-12-06T09:21:52.552 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:21:52.547+0000 13259640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:52.607 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:21:52.608+0000 13259640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:53.078 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:21:53.256 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:21:53.259+0000 bc92640 -1 osd.6 0 waiting for initial osdmap 2023-12-06T09:21:53.284 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:21:53.296+0000 b47e640 -1 osd.1 0 waiting for initial osdmap 2023-12-06T09:21:53.304 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:21:53.311+0000 b47e640 -1 osd.7 0 waiting for initial osdmap 2023-12-06T09:21:53.644 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:21:53.645+0000 13259640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:53.680 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:21:53.682+0000 13a5a640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:53.692 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:21:53.690+0000 1425b640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:53.702 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:21:53.702 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":13,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:52.790144+0000","last_up_change":"2023-12-06T09:21:52.790144+0000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:21:53.712 DEBUG:teuthology.misc:1 of 8 OSDs are up 2023-12-06T09:21:54.435 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:21:54.444+0000 bc92640 -1 osd.3 0 waiting for initial osdmap 2023-12-06T09:21:54.457 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:21:54.463+0000 bc92640 -1 osd.2 0 waiting for initial osdmap 2023-12-06T09:21:54.505 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:21:54.515+0000 b47e640 -1 osd.0 0 waiting for initial osdmap 2023-12-06T09:21:54.842 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:21:54.843+0000 1425b640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:54.852 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:21:54.851+0000 1425b640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:21:54.901 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:21:54.902+0000 1425b640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:22:00.520 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T09:22:01.234 INFO:teuthology.misc.health.smithi088.stdout: 2023-12-06T09:22:01.234 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":18,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:21:59.591103+0000","last_up_change":"2023-12-06T09:21:55.261571+0000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4165054852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":4165054852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":4165054852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4165054852}]},"public_addr":"172.21.15.132:6812/4165054852","cluster_addr":"172.21.15.132:6813/4165054852","heartbeat_back_addr":"172.21.15.132:6815/4165054852","heartbeat_front_addr":"172.21.15.132:6814/4165054852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:22:01.248 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-06T09:22:01.249 INFO:tasks.ceph:Creating RBD pool 2023-12-06T09:22:01.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-06T09:22:02.494 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'rbd' created 2023-12-06T09:22:02.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-06T09:22:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-06T09:22:03.376 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-06T09:22:03.377 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-06T09:22:03.377 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-06T09:22:03.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-06T09:22:03.377 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-06T09:22:04.299 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:04.312 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3185988515}]},"active_addr":"172.21.15.88:6800/3185988515","active_change":"2023-12-06T09:21:13.648401+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":2106693660}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":651229093}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":5809301}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":621441845}]}]} 2023-12-06T09:22:04.316 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-06T09:22:04.316 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-06T09:22:04.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:22:04.873 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:19.947 86645== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:04.929 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:20.003 86645== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:04.959 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:19.993 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:04.966 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:20.039 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:05.011 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:20.045 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:05.061 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:05.062 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:22:04.327786+0000","last_up_change":"2023-12-06T09:21:55.261571+0000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T09:22:01.832571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4165054852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":4165054852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":4165054852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4165054852}]},"public_addr":"172.21.15.132:6812/4165054852","cluster_addr":"172.21.15.132:6813/4165054852","heartbeat_back_addr":"172.21.15.132:6815/4165054852","heartbeat_front_addr":"172.21.15.132:6814/4165054852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:22:05.077 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-06T09:22:05.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:22:05.123 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:20.196 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:05.175 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:20.248 86645== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:05.246 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:20.280 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:05.370 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:20.443 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:05.969 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:05.969 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:22:04.327786+0000","last_up_change":"2023-12-06T09:21:55.261571+0000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T09:22:01.832571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4165054852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":4165054852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":4165054852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4165054852}]},"public_addr":"172.21.15.132:6812/4165054852","cluster_addr":"172.21.15.132:6813/4165054852","heartbeat_back_addr":"172.21.15.132:6815/4165054852","heartbeat_front_addr":"172.21.15.132:6814/4165054852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:22:05.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-06T09:22:05.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-06T09:22:05.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-06T09:22:05.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-06T09:22:05.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-06T09:22:05.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-06T09:22:05.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-06T09:22:05.985 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-06T09:22:06.328 INFO:teuthology.orchestra.run.smithi088.stdout:64424509444 2023-12-06T09:22:06.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T09:22:06.356 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:06.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T09:22:06.365 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:06.366 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T09:22:06.391 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-12-06T09:22:06.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T09:22:06.430 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:06.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T09:22:06.445 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:06.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T09:22:06.459 INFO:teuthology.orchestra.run.smithi088.stdout:64424509445 2023-12-06T09:22:06.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T09:22:06.526 INFO:teuthology.orchestra.run.smithi088.stdout:64424509446 2023-12-06T09:22:06.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T09:22:07.236 INFO:teuthology.orchestra.run.smithi088.stdout:60129542146 2023-12-06T09:22:07.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542146 for osd.7 2023-12-06T09:22:07.375 INFO:teuthology.orchestra.run.smithi088.stdout:60129542146 2023-12-06T09:22:07.378 INFO:teuthology.orchestra.run.smithi088.stdout:60129542146 2023-12-06T09:22:07.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542146 for osd.5 2023-12-06T09:22:07.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542146 for osd.1 2023-12-06T09:22:07.455 INFO:teuthology.orchestra.run.smithi088.stdout:64424509443 2023-12-06T09:22:07.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509443 for osd.3 2023-12-06T09:22:07.595 INFO:teuthology.orchestra.run.smithi088.stdout:64424509443 2023-12-06T09:22:07.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.2 2023-12-06T09:22:07.653 INFO:teuthology.orchestra.run.smithi088.stdout:60129542146 2023-12-06T09:22:07.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542146 for osd.6 2023-12-06T09:22:08.023 INFO:teuthology.orchestra.run.smithi088.stdout:55834574853 2023-12-06T09:22:08.024 INFO:teuthology.orchestra.run.smithi088.stdout:64424509444 2023-12-06T09:22:08.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.0 2023-12-06T09:22:08.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2023-12-06T09:22:08.255 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T09:22:08.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T09:22:08.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T09:22:08.473 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T09:22:08.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T09:22:08.667 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T09:22:09.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T09:22:09.041 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T09:22:09.132 INFO:teuthology.orchestra.run.smithi088.stdout:60129542147 2023-12-06T09:22:09.151 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.1 2023-12-06T09:22:09.152 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:09.168 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.7 2023-12-06T09:22:09.169 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:09.457 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:09.472 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.5 2023-12-06T09:22:09.472 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:09.500 INFO:teuthology.orchestra.run.smithi088.stdout:60129542148 2023-12-06T09:22:09.514 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.6 2023-12-06T09:22:09.514 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:09.551 INFO:teuthology.orchestra.run.smithi088.stdout:64424509445 2023-12-06T09:22:09.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2023-12-06T09:22:09.566 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:09.668 INFO:teuthology.orchestra.run.smithi088.stdout:64424509445 2023-12-06T09:22:09.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509445 for osd.2 2023-12-06T09:22:09.682 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:09.730 INFO:teuthology.orchestra.run.smithi088.stdout:64424509446 2023-12-06T09:22:09.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.0 2023-12-06T09:22:09.743 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:09.772 INFO:teuthology.orchestra.run.smithi088.stdout:55834574854 2023-12-06T09:22:09.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-12-06T09:22:09.785 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:10.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T09:22:10.774 INFO:teuthology.orchestra.run.smithi088.stdout:60129542149 2023-12-06T09:22:10.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.1 2023-12-06T09:22:10.787 DEBUG:teuthology.parallel:result is None 2023-12-06T09:22:10.788 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-06T09:22:10.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:22:11.339 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:11.339 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:22:11.352 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-12-06T09:22:09.922123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217648,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757072,"statfs":{"total":773094113280,"available":772871241728,"internally_reserved":0,"allocated":1515520,"data_stored":1284880,"data_compressed":1304,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":108,"apply_latency_ms":108,"commit_latency_ns":108000000,"apply_latency_ns":108000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":65,"num_write":82,"num_write_kb":1243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.666092"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.527253+0000","last_change":"2023-12-06T09:22:04.084324+0000","last_active":"2023-12-06T09:22:04.527253+0000","last_peered":"2023-12-06T09:22:04.527253+0000","last_clean":"2023-12-06T09:22:04.527253+0000","last_became_active":"2023-12-06T09:22:04.075169+0000","last_became_peered":"2023-12-06T09:22:04.075169+0000","last_unstale":"2023-12-06T09:22:04.527253+0000","last_undegraded":"2023-12-06T09:22:04.527253+0000","last_fullsized":"2023-12-06T09:22:04.527253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:06.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.773597+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:06.773597+0000","last_peered":"2023-12-06T09:22:06.773597+0000","last_clean":"2023-12-06T09:22:06.773597+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:06.773597+0000","last_undegraded":"2023-12-06T09:22:06.773597+0000","last_fullsized":"2023-12-06T09:22:06.773597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.523547+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:04.523547+0000","last_peered":"2023-12-06T09:22:04.523547+0000","last_clean":"2023-12-06T09:22:04.523547+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:04.523547+0000","last_undegraded":"2023-12-06T09:22:04.523547+0000","last_fullsized":"2023-12-06T09:22:04.523547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.515103+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:04.515103+0000","last_peered":"2023-12-06T09:22:04.515103+0000","last_clean":"2023-12-06T09:22:04.515103+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:04.515103+0000","last_undegraded":"2023-12-06T09:22:04.515103+0000","last_fullsized":"2023-12-06T09:22:04.515103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.518597+0000","last_change":"2023-12-06T09:22:04.156624+0000","last_active":"2023-12-06T09:22:04.518597+0000","last_peered":"2023-12-06T09:22:04.518597+0000","last_clean":"2023-12-06T09:22:04.518597+0000","last_became_active":"2023-12-06T09:22:04.151457+0000","last_became_peered":"2023-12-06T09:22:04.151457+0000","last_unstale":"2023-12-06T09:22:04.518597+0000","last_undegraded":"2023-12-06T09:22:04.518597+0000","last_fullsized":"2023-12-06T09:22:04.518597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:22:19.703097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.776388+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:06.776388+0000","last_peered":"2023-12-06T09:22:06.776388+0000","last_clean":"2023-12-06T09:22:06.776388+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:06.776388+0000","last_undegraded":"2023-12-06T09:22:06.776388+0000","last_fullsized":"2023-12-06T09:22:06.776388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.505390+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:04.505390+0000","last_peered":"2023-12-06T09:22:04.505390+0000","last_clean":"2023-12-06T09:22:04.505390+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:04.505390+0000","last_undegraded":"2023-12-06T09:22:04.505390+0000","last_fullsized":"2023-12-06T09:22:04.505390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.513059+0000","last_change":"2023-12-06T09:22:04.051938+0000","last_active":"2023-12-06T09:22:04.513059+0000","last_peered":"2023-12-06T09:22:04.513059+0000","last_clean":"2023-12-06T09:22:04.513059+0000","last_became_active":"2023-12-06T09:22:04.046554+0000","last_became_peered":"2023-12-06T09:22:04.046554+0000","last_unstale":"2023-12-06T09:22:04.513059+0000","last_undegraded":"2023-12-06T09:22:04.513059+0000","last_fullsized":"2023-12-06T09:22:04.513059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:51:40.895689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:08.710083+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:08.710083+0000","last_peered":"2023-12-06T09:22:08.710083+0000","last_clean":"2023-12-06T09:22:08.710083+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:08.710083+0000","last_undegraded":"2023-12-06T09:22:08.710083+0000","last_fullsized":"2023-12-06T09:22:08.710083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":49.691000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.388999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":42.523000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.939}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.707999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":166.45400000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.16}]}]},{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5039999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.364999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.295999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.202000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.966999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.123000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.702999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0069999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7420000000000009}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.492000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.826000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.542}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.797999999999998}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.196999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6719999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.276999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6520000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.295000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.097}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4189999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.060000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4089999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4480000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.054}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5149999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.517000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.42}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.956}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.789}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4109999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.894}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.929}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6509999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.644}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.148000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.605}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.73}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.393999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0839999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0410000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.588999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.353999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.076000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:22:11.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:22:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:11.936 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:22:11.949 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-12-06T09:22:11.922607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217648,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757072,"statfs":{"total":773094113280,"available":772871241728,"internally_reserved":0,"allocated":1515520,"data_stored":1284880,"data_compressed":1304,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":108,"apply_latency_ms":108,"commit_latency_ns":108000000,"apply_latency_ns":108000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":65,"num_write":82,"num_write_kb":1243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.541666"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.527253+0000","last_change":"2023-12-06T09:22:04.084324+0000","last_active":"2023-12-06T09:22:04.527253+0000","last_peered":"2023-12-06T09:22:04.527253+0000","last_clean":"2023-12-06T09:22:04.527253+0000","last_became_active":"2023-12-06T09:22:04.075169+0000","last_became_peered":"2023-12-06T09:22:04.075169+0000","last_unstale":"2023-12-06T09:22:04.527253+0000","last_undegraded":"2023-12-06T09:22:04.527253+0000","last_fullsized":"2023-12-06T09:22:04.527253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:06.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.773597+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:06.773597+0000","last_peered":"2023-12-06T09:22:06.773597+0000","last_clean":"2023-12-06T09:22:06.773597+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:06.773597+0000","last_undegraded":"2023-12-06T09:22:06.773597+0000","last_fullsized":"2023-12-06T09:22:06.773597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.523547+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:04.523547+0000","last_peered":"2023-12-06T09:22:04.523547+0000","last_clean":"2023-12-06T09:22:04.523547+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:04.523547+0000","last_undegraded":"2023-12-06T09:22:04.523547+0000","last_fullsized":"2023-12-06T09:22:04.523547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.515103+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:04.515103+0000","last_peered":"2023-12-06T09:22:04.515103+0000","last_clean":"2023-12-06T09:22:04.515103+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:04.515103+0000","last_undegraded":"2023-12-06T09:22:04.515103+0000","last_fullsized":"2023-12-06T09:22:04.515103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.518597+0000","last_change":"2023-12-06T09:22:04.156624+0000","last_active":"2023-12-06T09:22:04.518597+0000","last_peered":"2023-12-06T09:22:04.518597+0000","last_clean":"2023-12-06T09:22:04.518597+0000","last_became_active":"2023-12-06T09:22:04.151457+0000","last_became_peered":"2023-12-06T09:22:04.151457+0000","last_unstale":"2023-12-06T09:22:04.518597+0000","last_undegraded":"2023-12-06T09:22:04.518597+0000","last_fullsized":"2023-12-06T09:22:04.518597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:22:19.703097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.776388+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:06.776388+0000","last_peered":"2023-12-06T09:22:06.776388+0000","last_clean":"2023-12-06T09:22:06.776388+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:06.776388+0000","last_undegraded":"2023-12-06T09:22:06.776388+0000","last_fullsized":"2023-12-06T09:22:06.776388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.505390+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:04.505390+0000","last_peered":"2023-12-06T09:22:04.505390+0000","last_clean":"2023-12-06T09:22:04.505390+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:04.505390+0000","last_undegraded":"2023-12-06T09:22:04.505390+0000","last_fullsized":"2023-12-06T09:22:04.505390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.513059+0000","last_change":"2023-12-06T09:22:04.051938+0000","last_active":"2023-12-06T09:22:04.513059+0000","last_peered":"2023-12-06T09:22:04.513059+0000","last_clean":"2023-12-06T09:22:04.513059+0000","last_became_active":"2023-12-06T09:22:04.046554+0000","last_became_peered":"2023-12-06T09:22:04.046554+0000","last_unstale":"2023-12-06T09:22:04.513059+0000","last_undegraded":"2023-12-06T09:22:04.513059+0000","last_fullsized":"2023-12-06T09:22:04.513059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:51:40.895689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:08.710083+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:08.710083+0000","last_peered":"2023-12-06T09:22:08.710083+0000","last_clean":"2023-12-06T09:22:08.710083+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:08.710083+0000","last_undegraded":"2023-12-06T09:22:08.710083+0000","last_fullsized":"2023-12-06T09:22:08.710083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9740000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.071999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.618}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.274000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.103999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.103000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.975999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9369999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.66}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4269999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.225999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.414999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.423999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.465999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4039999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8789999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.425000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.812999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.786999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.196999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6719999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.276999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6520000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.295000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.097}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4189999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.060000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4089999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4480000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.054}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5149999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.517000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.42}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.956}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.789}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4109999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.894}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.929}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6509999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.644}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.148000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.605}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.73}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.393999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0839999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0410000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.588999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.353999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.076000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:22:11.950 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-06T09:22:11.950 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-06T09:22:11.950 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-06T09:22:11.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-06T09:22:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:12.541 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-06T09:22:12.555 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-06T09:22:12.555 INFO:teuthology.run_tasks:Running task thrashosds... 2023-12-06T09:22:12.566 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-12-06T09:22:12.567 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sseshasa-testing-2023-12-04-1848'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-sseshasa-testing-2023-12-04-1848', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}} 2023-12-06T09:22:12.567 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2023-12-06T09:22:12.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-12-06T09:22:13.268 INFO:tasks.thrashosds:Beginning thrashosds... 2023-12-06T09:22:13.268 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-06T09:22:13.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:22:13.810 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:13.810 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:22:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-12-06T09:22:11.922607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217648,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757072,"statfs":{"total":773094113280,"available":772871241728,"internally_reserved":0,"allocated":1515520,"data_stored":1284880,"data_compressed":1304,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":108,"apply_latency_ms":108,"commit_latency_ns":108000000,"apply_latency_ns":108000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":65,"num_write":82,"num_write_kb":1243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.541666"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.527253+0000","last_change":"2023-12-06T09:22:04.084324+0000","last_active":"2023-12-06T09:22:04.527253+0000","last_peered":"2023-12-06T09:22:04.527253+0000","last_clean":"2023-12-06T09:22:04.527253+0000","last_became_active":"2023-12-06T09:22:04.075169+0000","last_became_peered":"2023-12-06T09:22:04.075169+0000","last_unstale":"2023-12-06T09:22:04.527253+0000","last_undegraded":"2023-12-06T09:22:04.527253+0000","last_fullsized":"2023-12-06T09:22:04.527253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:06.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.773597+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:06.773597+0000","last_peered":"2023-12-06T09:22:06.773597+0000","last_clean":"2023-12-06T09:22:06.773597+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:06.773597+0000","last_undegraded":"2023-12-06T09:22:06.773597+0000","last_fullsized":"2023-12-06T09:22:06.773597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.523547+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:04.523547+0000","last_peered":"2023-12-06T09:22:04.523547+0000","last_clean":"2023-12-06T09:22:04.523547+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:04.523547+0000","last_undegraded":"2023-12-06T09:22:04.523547+0000","last_fullsized":"2023-12-06T09:22:04.523547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.515103+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:04.515103+0000","last_peered":"2023-12-06T09:22:04.515103+0000","last_clean":"2023-12-06T09:22:04.515103+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:04.515103+0000","last_undegraded":"2023-12-06T09:22:04.515103+0000","last_fullsized":"2023-12-06T09:22:04.515103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.518597+0000","last_change":"2023-12-06T09:22:04.156624+0000","last_active":"2023-12-06T09:22:04.518597+0000","last_peered":"2023-12-06T09:22:04.518597+0000","last_clean":"2023-12-06T09:22:04.518597+0000","last_became_active":"2023-12-06T09:22:04.151457+0000","last_became_peered":"2023-12-06T09:22:04.151457+0000","last_unstale":"2023-12-06T09:22:04.518597+0000","last_undegraded":"2023-12-06T09:22:04.518597+0000","last_fullsized":"2023-12-06T09:22:04.518597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:22:19.703097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.776388+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:06.776388+0000","last_peered":"2023-12-06T09:22:06.776388+0000","last_clean":"2023-12-06T09:22:06.776388+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:06.776388+0000","last_undegraded":"2023-12-06T09:22:06.776388+0000","last_fullsized":"2023-12-06T09:22:06.776388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.505390+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:04.505390+0000","last_peered":"2023-12-06T09:22:04.505390+0000","last_clean":"2023-12-06T09:22:04.505390+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:04.505390+0000","last_undegraded":"2023-12-06T09:22:04.505390+0000","last_fullsized":"2023-12-06T09:22:04.505390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.513059+0000","last_change":"2023-12-06T09:22:04.051938+0000","last_active":"2023-12-06T09:22:04.513059+0000","last_peered":"2023-12-06T09:22:04.513059+0000","last_clean":"2023-12-06T09:22:04.513059+0000","last_became_active":"2023-12-06T09:22:04.046554+0000","last_became_peered":"2023-12-06T09:22:04.046554+0000","last_unstale":"2023-12-06T09:22:04.513059+0000","last_undegraded":"2023-12-06T09:22:04.513059+0000","last_fullsized":"2023-12-06T09:22:04.513059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:51:40.895689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:08.710083+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:08.710083+0000","last_peered":"2023-12-06T09:22:08.710083+0000","last_clean":"2023-12-06T09:22:08.710083+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:08.710083+0000","last_undegraded":"2023-12-06T09:22:08.710083+0000","last_fullsized":"2023-12-06T09:22:08.710083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9740000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.071999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.618}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.274000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.103999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.103000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.975999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9369999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.66}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4269999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.225999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.414999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.423999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.465999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4039999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8789999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.425000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.812999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.786999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.196999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6719999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.276999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6520000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.295000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.097}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4189999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.060000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4089999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4480000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.054}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5149999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.517000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.42}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.956}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.789}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4109999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.894}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.929}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6509999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.644}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.148000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.605}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.73}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.393999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0839999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0410000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.588999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.353999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.076000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:22:13.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:22:14.406 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:14.407 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:22:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-12-06T09:22:13.922908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217648,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757072,"statfs":{"total":773094113280,"available":772871241728,"internally_reserved":0,"allocated":1515520,"data_stored":1284880,"data_compressed":1304,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":108,"apply_latency_ms":108,"commit_latency_ns":108000000,"apply_latency_ns":108000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":65,"num_write":82,"num_write_kb":1243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.527253+0000","last_change":"2023-12-06T09:22:04.084324+0000","last_active":"2023-12-06T09:22:04.527253+0000","last_peered":"2023-12-06T09:22:04.527253+0000","last_clean":"2023-12-06T09:22:04.527253+0000","last_became_active":"2023-12-06T09:22:04.075169+0000","last_became_peered":"2023-12-06T09:22:04.075169+0000","last_unstale":"2023-12-06T09:22:04.527253+0000","last_undegraded":"2023-12-06T09:22:04.527253+0000","last_fullsized":"2023-12-06T09:22:04.527253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:06.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.773597+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:06.773597+0000","last_peered":"2023-12-06T09:22:06.773597+0000","last_clean":"2023-12-06T09:22:06.773597+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:06.773597+0000","last_undegraded":"2023-12-06T09:22:06.773597+0000","last_fullsized":"2023-12-06T09:22:06.773597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.523547+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:04.523547+0000","last_peered":"2023-12-06T09:22:04.523547+0000","last_clean":"2023-12-06T09:22:04.523547+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:04.523547+0000","last_undegraded":"2023-12-06T09:22:04.523547+0000","last_fullsized":"2023-12-06T09:22:04.523547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.515103+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:04.515103+0000","last_peered":"2023-12-06T09:22:04.515103+0000","last_clean":"2023-12-06T09:22:04.515103+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:04.515103+0000","last_undegraded":"2023-12-06T09:22:04.515103+0000","last_fullsized":"2023-12-06T09:22:04.515103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.518597+0000","last_change":"2023-12-06T09:22:04.156624+0000","last_active":"2023-12-06T09:22:04.518597+0000","last_peered":"2023-12-06T09:22:04.518597+0000","last_clean":"2023-12-06T09:22:04.518597+0000","last_became_active":"2023-12-06T09:22:04.151457+0000","last_became_peered":"2023-12-06T09:22:04.151457+0000","last_unstale":"2023-12-06T09:22:04.518597+0000","last_undegraded":"2023-12-06T09:22:04.518597+0000","last_fullsized":"2023-12-06T09:22:04.518597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:22:19.703097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.776388+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:06.776388+0000","last_peered":"2023-12-06T09:22:06.776388+0000","last_clean":"2023-12-06T09:22:06.776388+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:06.776388+0000","last_undegraded":"2023-12-06T09:22:06.776388+0000","last_fullsized":"2023-12-06T09:22:06.776388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.505390+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:04.505390+0000","last_peered":"2023-12-06T09:22:04.505390+0000","last_clean":"2023-12-06T09:22:04.505390+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:04.505390+0000","last_undegraded":"2023-12-06T09:22:04.505390+0000","last_fullsized":"2023-12-06T09:22:04.505390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.513059+0000","last_change":"2023-12-06T09:22:04.051938+0000","last_active":"2023-12-06T09:22:04.513059+0000","last_peered":"2023-12-06T09:22:04.513059+0000","last_clean":"2023-12-06T09:22:04.513059+0000","last_became_active":"2023-12-06T09:22:04.046554+0000","last_became_peered":"2023-12-06T09:22:04.046554+0000","last_unstale":"2023-12-06T09:22:04.513059+0000","last_undegraded":"2023-12-06T09:22:04.513059+0000","last_fullsized":"2023-12-06T09:22:04.513059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:51:40.895689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:08.710083+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:08.710083+0000","last_peered":"2023-12-06T09:22:08.710083+0000","last_clean":"2023-12-06T09:22:08.710083+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:08.710083+0000","last_undegraded":"2023-12-06T09:22:08.710083+0000","last_fullsized":"2023-12-06T09:22:08.710083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9740000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.071999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.618}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.274000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.103999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.103000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.975999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9369999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.66}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4269999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.225999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.414999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.423999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.465999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4039999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8789999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.425000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.812999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.786999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.196999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6719999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.276999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6520000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.295000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.097}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4189999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.060000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4089999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4480000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.054}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5149999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.517000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.42}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.956}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.789}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4109999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.894}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":458990,"data_compressed":652,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.929}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6509999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.644}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.148000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.605}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.73}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.393999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":61150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0839999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0410000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.588999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.353999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.076000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:22:14.421 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-06T09:22:14.421 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-12-06T09:22:15.041 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 21 2023-12-06T09:22:15.041 INFO:teuthology.orchestra.run.smithi088.stdout:fsid 0c48cca3-dcc0-44bd-8eed-37fea2138550 2023-12-06T09:22:15.041 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-12-06T09:21:08.071156+0000 2023-12-06T09:22:15.041 INFO:teuthology.orchestra.run.smithi088.stdout:modified 2023-12-06T09:22:04.327786+0000 2023-12-06T09:22:15.041 INFO:teuthology.orchestra.run.smithi088.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-12-06T09:22:15.041 INFO:teuthology.orchestra.run.smithi088.stdout:crush_version 6 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:full_ratio 0.95 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:backfillfull_ratio 0.9 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:nearfull_ratio 0.85 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:require_min_compat_client luminous 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:min_compat_client jewel 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:require_osd_release reef 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:stretch_mode_enabled false 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 application rbd read_balance_score 4.98 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:max_osd 8 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6805/2388339328 v2:172.21.15.88:6806/2388339328 exists,up db81b709-a318-4b43-beb2-bf3687f365e1 2023-12-06T09:22:15.042 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1 up in weight 1 up_from 14 up_thru 15 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6801/3646708602 v2:172.21.15.88:6802/3646708602 exists,up fd7b0482-e237-424b-8eff-8db905713049 2023-12-06T09:22:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6810/3714168299 v2:172.21.15.88:6813/3714168299 exists,up e5a571be-6b8e-4c6f-bd3e-2af30c112285 2023-12-06T09:22:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6809/1828251745 v2:172.21.15.88:6811/1828251745 exists,up d7abe19a-cc3d-411f-8533-8592e1379735 2023-12-06T09:22:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6800/3720978101 v2:172.21.15.132:6801/3720978101 exists,up b9517160-d081-44f9-aea4-6d2529e02114 2023-12-06T09:22:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6804/1203998358 v2:172.21.15.132:6805/1203998358 exists,up b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e 2023-12-06T09:22:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6808/4151455937 v2:172.21.15.132:6809/4151455937 exists,up e7afeca8-1363-4e9b-96f6-7fab6113d9e0 2023-12-06T09:22:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6812/4165054852 v2:172.21.15.132:6813/4165054852 exists,up fd4b3971-3adb-44fd-b42b-b6660305beb3 2023-12-06T09:22:15.056 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6805/2388339328 v2:172.21.15.88:6806/2388339328 exists,up db81b709-a318-4b43-beb2-bf3687f365e1', 'osd.1 up in weight 1 up_from 14 up_thru 15 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6801/3646708602 v2:172.21.15.88:6802/3646708602 exists,up fd7b0482-e237-424b-8eff-8db905713049', 'osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6810/3714168299 v2:172.21.15.88:6813/3714168299 exists,up e5a571be-6b8e-4c6f-bd3e-2af30c112285', 'osd.3 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.88:6809/1828251745 v2:172.21.15.88:6811/1828251745 exists,up d7abe19a-cc3d-411f-8533-8592e1379735', 'osd.4 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6800/3720978101 v2:172.21.15.132:6801/3720978101 exists,up b9517160-d081-44f9-aea4-6d2529e02114', 'osd.5 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6804/1203998358 v2:172.21.15.132:6805/1203998358 exists,up b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e', 'osd.6 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6808/4151455937 v2:172.21.15.132:6809/4151455937 exists,up e7afeca8-1363-4e9b-96f6-7fab6113d9e0', 'osd.7 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.132:6812/4165054852 v2:172.21.15.132:6813/4165054852 exists,up fd4b3971-3adb-44fd-b42b-b6660305beb3'] 2023-12-06T09:22:15.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "mon.a", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster": "ceph", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "admin_socket_mode": "", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_client_required": "cephx, none", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_cluster_required": "cephx", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_debug": "true", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-12-06T09:22:15.237 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_service_required": "cephx", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_service_ticket_ttl": "240.000000", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_supported": "cephx", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_aio": "true", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_aio_max_queue_depth": "1024", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_aio_poll_ms": "250", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_aio_reap_max": "16", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_async_discard": "false", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_block_size": "4096", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_debug_aio": "false", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-12-06T09:22:15.238 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_debug_inflight_ios": "false", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_enable_discard": "false", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_flock_retry": "3", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_flock_retry_interval": "0.100000", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_inject_crash": "0", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_inject_crash_flush_delay": "2", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_ioring": "false", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_ioring_hipri": "false", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_ioring_sqthread_poll": "false", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_read_buffer_alignment": "4096", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bdev_type": "", 2023-12-06T09:22:15.239 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_alloc_size": "1048576", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_allocator": "hybrid", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_buffered_io": "true", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_check_for_zeros": "false", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_check_volume_selector_often": "false", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_compact_log_sync": "false", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_log_compact_min_size": "16777216", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_log_replay_check_allocations": "true", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_max_log_runway": "4194304", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_max_prefetch": "1048576", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_min_flush_size": "524288", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_min_log_runway": "1048576", 2023-12-06T09:22:15.240 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_replay_recovery": "false", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_shared_alloc_size": "65536", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_sync_write": "false", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_allocation_from_file": "true", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_allocator": "hybrid", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-12-06T09:22:15.241 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blobid_prealloc": "10240", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_create": "true", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_db_create": "false", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_db_path": "", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_db_size": "0", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_path": "", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_preallocate_file": "false", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_size": "107374182400", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_wal_create": "false", 2023-12-06T09:22:15.242 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_wal_path": "", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_block_wal_size": "100663296", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_bluefs": "true", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_bluefs_env_mirror": "false", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_autotune": "true", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-12-06T09:22:15.243 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_size": "0", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_type": "2q", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_clone_cow": "true", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_algorithm": "snappy", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_max_blob_size": "0", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-12-06T09:22:15.244 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_min_blob_size": "0", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_mode": "none", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_csum_type": "crc32c", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_enforce_settings": "default", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_freelist": "false", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_fsck_abort": "false", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_inject_read_err": "false", 2023-12-06T09:22:15.245 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_legacy_omap": "false", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_prefill": "0.000000", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_small_allocations": "0", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_default_buffered_read": "true", 2023-12-06T09:22:15.246 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_default_buffered_write": "false", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_deferred_batch_ops": "0", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_elastic_shared_blobs": "true", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fail_eio": "false", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-12-06T09:22:15.247 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_on_mkfs": "true", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_on_mount": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_on_umount": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-12-06T09:22:15.248 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_ignore_data_csum": "false", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_kvbackend": "rocksdb", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_log_op_age": "5.000000", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_max_alloc_size": "0", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_max_blob_size": "0", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-12-06T09:22:15.249 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_max_defer_interval": "3.000000", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_max_deferred_txc": "32", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_min_alloc_size": "0", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_nid_prealloc": "1024", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_prefer_deferred_size": "0", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_qfsck_on_mount": "true", 2023-12-06T09:22:15.250 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_retry_disk_reads": "3", 2023-12-06T09:22:15.254 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_rocksdb_cf": "true", 2023-12-06T09:22:15.254 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-12-06T09:22:15.254 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-12-06T09:22:15.254 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_rocksdb_options_annex": "", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_spdk_coremask": "0x1", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_spdk_io_sleep": "5", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_spdk_max_io_completion": "0", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_spdk_mem": "512", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_sync_submit_transaction": "false", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_throttle_bytes": "67108864", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_throttle_cost_per_io": "0", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_tracing": "false", 2023-12-06T09:22:15.255 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_volume_selection_reserved": "0", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_zero_block_detection": "false", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_action_update_interval": "2", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-12-06T09:22:15.256 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_mount_timeout": "10", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_cluster_require_signatures": "false", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_cluster_require_version": "2", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_require_signatures": "false", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_require_version": "2", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_service_require_signatures": "false", 2023-12-06T09:22:15.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_service_require_version": "2", 2023-12-06T09:22:15.258 INFO:teuthology.orchestra.run.smithi088.stdout: "cephx_sign_messages": "true", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "chdir": "", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_acl_type": "", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_asio_thread_count": "2", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_cache_mid": "0.750000", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_cache_size": "16384", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_caps_release_delay": "5", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_check_pool_perm": "true", 2023-12-06T09:22:15.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_collect_and_send_global_metrics": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_debug_force_sync_read": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_debug_getattr_caps": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_debug_inject_tick_delay": "0", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_die_on_failed_remount": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_dirsize_rbytes": "true", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_force_lazyio": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_fs": "", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_inject_fixed_oldest_tid": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_inject_release_failure": "false", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_max_inline_size": "4096", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_max_retries_on_remount_failure": "5", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_mds_namespace": "", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_metadata": "", 2023-12-06T09:22:15.260 INFO:teuthology.orchestra.run.smithi088.stdout: "client_mount_gid": "-1", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_mount_timeout": "300", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_mount_uid": "-1", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_mountpoint": "/", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_notify_timeout": "10", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_oc": "true", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_oc_max_dirty": "104857600", 2023-12-06T09:22:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_oc_max_dirty_age": "5.000000", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_oc_max_objects": "1000", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_oc_size": "209715200", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_oc_target_dirty": "8388608", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_permissions": "true", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_quota": "true", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_quota_df": "true", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_readahead_max_bytes": "0", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_readahead_max_periods": "4", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_readahead_min": "131072", 2023-12-06T09:22:15.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_reconnect_stale": "false", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_shutdown_timeout": "30", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_snapdir": ".snap", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_tick_interval": "1", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_trace": "", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_try_dentry_invalidate": "false", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_use_faked_inos": "false", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_use_random_mds": "false", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_graylog": "false", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_graylog_port": "12201", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_monitors": "default=true", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_syslog": "false", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "clog_to_syslog_level": "info", 2023-12-06T09:22:15.263 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "-", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_network": "", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_network_interface": "", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "compressor_zlib_isal": "false", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "compressor_zlib_level": "5", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "compressor_zlib_winsize": "-15", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "compressor_zstd_level": "1", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crimson_alien_op_num_threads": "6", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crimson_alien_thread_cpu_cores": "", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crimson_osd_obc_lru_size": "10", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crimson_seastar_smp": "1", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "", 2023-12-06T09:22:15.264 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location_hook": "", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location_hook_timeout": "10", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "daemonize": "false", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "daos_pool": "tank", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "dbstore_db_name_prefix": "dbstore", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_alienstore": "0/5", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_allow_any_pool_priority": "false", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_asok": "1/5", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_asok_assert_abort": "false", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_asserts_on_shutdown": "true", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_auth": "20/20", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_bdev": "1/3", 2023-12-06T09:22:15.265 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_bluefs": "1/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_bluestore": "1/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_buffer": "0/1", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_ceph_exporter": "1/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_cephfs_mirror": "0/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_cephsqlite": "0/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_client": "0/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_compressor": "1/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_context": "0/1", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_crimson_interrupt": "0/5", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_crush": "1/1", 2023-12-06T09:22:15.266 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_crypto": "1/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_cyanstore": "0/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_deliberately_leak_memory": "false", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_disable_randomized_ping": "false", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_dpdk": "1/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_eventtrace": "1/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_filer": "0/1", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_filestore": "1/3", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_finisher": "1/1", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_fuse": "1/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_heartbeat_testing_span": "0", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_heartbeatmap": "1/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_immutable_obj_cache": "0/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_javaclient": "1/5", 2023-12-06T09:22:15.267 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_journal": "1/3", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_journaler": "0/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_kstore": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_lockdep": "0/1", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mclock": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mds": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mds_balancer": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mds_locker": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mds_log": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mds_log_expire": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mds_migrator": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_memstore": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mgr": "2/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mgrc": "1/5", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_mon": "20/20", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_monc": "0/10", 2023-12-06T09:22:15.268 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_ms": "1/1", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_none": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_objclass": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_objectcacher": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_objecter": "0/1", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_optracker": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_osd": "1/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_paxos": "20/20", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_perfcounter": "1/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_prioritycache": "1/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rados": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rbd": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rbd_mirror": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rbd_pwl": "0/5", 2023-12-06T09:22:15.269 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rbd_replay": "0/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_refs": "0/0", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_reserver": "1/1", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rgw": "1/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rgw_access": "1/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rgw_datacache": "1/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rgw_dbstore": "1/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rgw_flight": "1/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rgw_sync": "1/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_rocksdb": "4/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore": "0/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_backref": "0/5", 2023-12-06T09:22:15.270 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_cache": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_cleaner": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_device": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_epm": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_journal": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_lba": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_odata": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_omap": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_onode": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_t": "0/5", 2023-12-06T09:22:15.271 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_seastore_tm": "0/5", 2023-12-06T09:22:15.272 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_striper": "0/1", 2023-12-06T09:22:15.272 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_test": "0/5", 2023-12-06T09:22:15.272 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_throttle": "1/1", 2023-12-06T09:22:15.272 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_timer": "0/1", 2023-12-06T09:22:15.273 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_tp": "0/5", 2023-12-06T09:22:15.274 INFO:teuthology.orchestra.run.smithi088.stdout: "debug_trace": "1/5", 2023-12-06T09:22:15.274 INFO:teuthology.orchestra.run.smithi088.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-12-06T09:22:15.274 INFO:teuthology.orchestra.run.smithi088.stdout: "device_failure_prediction_mode": "none", 2023-12-06T09:22:15.274 INFO:teuthology.orchestra.run.smithi088.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-12-06T09:22:15.274 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2023-12-06T09:22:15.274 INFO:teuthology.orchestra.run.smithi088.stdout: "err_to_graylog": "false", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "err_to_journald": "false", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "err_to_stderr": "true", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "err_to_syslog": "false", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event_tracing": "false", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "exporter_addr": "0.0.0.0", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "exporter_http_port": "9926", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "exporter_prio_limit": "5", 2023-12-06T09:22:15.275 INFO:teuthology.orchestra.run.smithi088.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "exporter_sort_metrics": "true", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "exporter_stats_period": "5", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "fake_statfs_for_testing": "0", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "fatal_signal_handlers": "true", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filer_max_purge_ops": "10", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filer_max_truncate_ops": "128", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_apply_finisher_threads": "1", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_blackhole": "false", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_btrfs_clone_range": "true", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_btrfs_snap": "true", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_caller_concurrency": "10", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_collect_device_partition_information": "true", 2023-12-06T09:22:15.276 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_commit_timeout": "600.000000", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_debug_inject_read_err": "false", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_debug_omap_check": "false", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_debug_random_read_err": "0.000000", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_debug_verify_split": "false", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_dump_file": "", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fadvise": "true", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fail_eio": "true", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fd_cache_shards": "16", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fd_cache_size": "128", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fiemap": "false", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fiemap_threshold": "4096", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-12-06T09:22:15.277 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_index_retry_probability": "0.000000", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_inject_stall": "0", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_journal_parallel": "false", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_journal_trailing": "false", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_journal_writeahead": "false", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_kill_at": "0", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattr_size": "0", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattrs": "0", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattrs_other": "2", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-12-06T09:22:15.278 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_sync_interval": "5.000000", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_xattr_value_size": "0", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_merge_threshold": "-10", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_min_sync_interval": "0.010000", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_odsync_write": "false", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_omap_backend": "rocksdb", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_omap_backend_path": "", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_omap_header_cache_size": "1024", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_ondisk_finisher_threads": "1", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_op_thread_timeout": "60", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_op_threads": "2", 2023-12-06T09:22:15.279 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_punch_hole": "false", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_max_bytes": "104857600", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_queue_max_ops": "50", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_seek_data_hole": "false", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_sloppy_crc": "false", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_splice": "false", 2023-12-06T09:22:15.280 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_split_multiple": "2", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_split_rand_factor": "20", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_update_to": "1000", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_enable": "true", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-12-06T09:22:15.281 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_xfs_extsize": "false", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "filestore_zfs_snap": "false", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fio_dir": "/tmp/fio", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_allow_other": "true", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_atomic_o_trunc": "true", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_big_writes": "true", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_debug": "false", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_default_permissions": "false", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_disable_pagecache": "false", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_max_write": "0", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_multithreaded": "true", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_require_active_mds": "true", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_set_user_groups": "true", 2023-12-06T09:22:15.282 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_splice_move": "true", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_splice_read": "true", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_splice_write": "true", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_syncfs_on_mksnap": "true", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "fuse_use_invalidate_cb": "true", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "gss_target_name": "ceph", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_file": "", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_inject_failure": "0", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_interval": "5", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "host": "smithi088", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_max_size": "1073741824", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-12-06T09:22:15.283 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "immutable_object_cache_watermark": "0.900000", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "inject_early_sigterm": "false", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "jaeger_agent_port": "6799", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "jaeger_tracing_enable": "false", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_aio": "true", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_align_min_size": "65536", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_block_align": "true", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_block_size": "4096", 2023-12-06T09:22:15.284 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_dio": "true", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_discard": "false", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_force_aio": "false", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_ignore_corruption": "false", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_max_write_bytes": "10485760", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_max_write_entries": "100", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_replay_from": "0", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_throttle_high_multiple": "0.000000", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_throttle_max_multiple": "0.000000", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_write_header_frequency": "0", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journal_zero_on_create": "false", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journaler_prefetch_periods": "10", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journaler_prezero_periods": "5", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "journaler_write_head_interval": "15", 2023-12-06T09:22:15.285 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "keyfile": "", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_backend": "rocksdb", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_default_stripe_size": "65536", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_fsck_on_mount": "false", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_fsck_on_mount_deep": "true", 2023-12-06T09:22:15.286 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_max_bytes": "67108864", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_max_ops": "512", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_nid_prealloc": "1024", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_onode_map_size": "1024", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_sync_submit_transaction": "false", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "kstore_sync_transaction": "false", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "librados_thread_count": "2", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "lockdep": "false", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "lockdep_force_backtrace": "false", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "log_coarse_timestamps": "true", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "log_flush_on_exit": "false", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "log_graylog_host": "127.0.0.1", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "log_graylog_port": "12201", 2023-12-06T09:22:15.287 INFO:teuthology.orchestra.run.smithi088.stdout: "log_max_new": "1000", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_max_recent": "10000", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_stderr_prefix": "", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_stop_at_utilization": "0.970000", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_to_file": "true", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_to_graylog": "false", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_to_journald": "false", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_to_stderr": "false", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "log_to_syslog": "false", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "mandatory_topic_permissions": "false", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "max_rotating_auth_attempts": "10", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_action_on_write_error": "1", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_alternate_name_max": "8192", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_asio_thread_count": "2", 2023-12-06T09:22:15.288 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_export_pin": "true", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_fragment_dirs": "true", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_fragment_interval": "5", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_fragment_size_max": "100000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_idle_threshold": "0.000000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_interval": "10", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_max": "-1", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_max_until": "-1", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_merge_size": "50", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_midchunk": "0.300000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_min_rebalance": "0.100000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_min_start": "0.200000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_minchunk": "0.001000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_mode": "0", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_need_max": "1.200000", 2023-12-06T09:22:15.289 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_need_min": "0.800000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_overload_epochs": "2", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_sample_interval": "3.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_split_bits": "3", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_split_rd": "25000.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_split_size": "10000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_split_wr": "10000.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_target_decay": "10.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_beacon_grace": "15.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_beacon_interval": "4.000000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_beacon_mon_down_grace": "60", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_memory_limit": "4294967296", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_mid": "0.700000", 2023-12-06T09:22:15.290 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_release_free_interval": "10", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_reservation": "0.050000", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_trim_interval": "1", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache_trim_threshold": "262144", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_client_delegate_inos_pct": "50", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_client_prealloc_inos": "1000", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_connect_bootstrapping": "false", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_damage_table_max_entries": "10000", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_debug_auth_pins": "false", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_debug_frag": "false", 2023-12-06T09:22:15.291 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_debug_scatterstat": "false", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_debug_subtrees": "false", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_decay_halflife": "5.000000", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_default_dir_hash": "2", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_defer_session_stale": "true", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_deny_all_reconnect": "false", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dir_keys_per_op": "16384", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dir_max_commit_size": "10", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dir_max_entries": "0", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dir_prefetch": "true", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dirstat_min_interval": "1.000000", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dump_cache_after_rejoin": "false", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dump_cache_on_map": "false", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dump_cache_threshold_file": "0", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_early_reply": "true", 2023-12-06T09:22:15.292 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_enable_op_tracker": "true", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_enforce_unique_name": "true", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_export_ephemeral_distributed": "true", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_export_ephemeral_random": "true", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_forward_all_requests_to_auth": "false", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_health_cache_threshold": "1.500000", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_health_summarize_threshold": "10", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_heartbeat_grace": "15.000000", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_heartbeat_reset_grace": "1000", 2023-12-06T09:22:15.293 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_inject_health_dummy": "false", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_inject_migrator_session_race": "false", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_join_fs": "", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_journal_format": "1", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_create_at": "0", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_export_at": "0", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_import_at": "0", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_journal_at": "0", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_journal_expire_at": "0", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_journal_replay_at": "0", 2023-12-06T09:22:15.294 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_link_at": "0", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_mdstable_at": "0", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_openc_at": "0", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_rename_at": "0", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_shutdown_at": "0", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_event_large_threshold": "524288", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_events_per_segment": "1024", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_major_segment_event_ratio": "12", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_max_events": "-1", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_max_segments": "128", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_pause": "false", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_segment_size": "0", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_skip_corrupt_events": "false", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_skip_unbounded_events": "false", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log_warn_factor": "2.000000", 2023-12-06T09:22:15.295 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_caps_per_client": "1048576", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_completed_flushes": "100000", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_completed_requests": "100000", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_export_size": "20971520", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_file_recover": "32", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_purge_files": "64", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_purge_ops": "8192", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-12-06T09:22:15.296 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_max_snaps_per_dir": "100", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_metrics_update_interval": "2", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_min_caps_per_client": "100", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_min_caps_working_set": "10000", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_numa_node": "-1", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_op_complaint_time": "30.000000", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_op_history_duration": "600", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_op_history_size": "20", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_op_history_slow_op_size": "20", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_op_history_slow_op_threshold": "10", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_op_log_threshold": "5", 2023-12-06T09:22:15.297 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_ping_grace": "15", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_ping_interval": "5", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_recall_max_caps": "30000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_recall_max_decay_threshold": "131072", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_recall_warning_threshold": "262144", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_reconnect_timeout": "45.000000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_replay_interval": "1.000000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_root_ino_gid": "0", 2023-12-06T09:22:15.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_root_ino_uid": "0", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_blocklist_on_evict": "true", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_blocklist_on_timeout": "true", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_session_metadata_threshold": "16777216", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_shutdown_check": "0", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_skip_ino": "0", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_sleep_rank_change": "0.000000", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_snap_max_uid": "4294967294", 2023-12-06T09:22:15.299 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_snap_min_uid": "0", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_snap_rstat": "false", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_standby_replay_damaged": "false", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_symlink_recovery": "true", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_task_status_update_interval": "2.000000", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_thrash_exports": "0", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_thrash_fragments": "0", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_tick_interval": "5.000000", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_valgrind_exit": "false", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_verify_backtrace": "1", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_verify_scatter": "false", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_wipe_ino_prealloc": "false", 2023-12-06T09:22:15.300 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_wipe_sessions": "false", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mempool_debug": "false", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "memstore_debug_omit_block_device_write": "false", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "memstore_device_bytes": "1073741824", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "memstore_page_set": "false", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "memstore_page_size": "65536", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_client_bytes": "134217728", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_client_messages": "512", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_connect_retry_interval": "1.000000", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-12-06T09:22:15.301 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_disabled_modules": "", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_max_pg_num_change": "128", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_mds_bytes": "134217728", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_mds_messages": "128", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_mon_bytes": "134217728", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_mon_messages": "128", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_osd_bytes": "536870912", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_osd_messages": "8192", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_pool": "true", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_service_beacon_grace": "60.000000", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_standby_modules": "true", 2023-12-06T09:22:15.302 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_stats_period": "5", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_stats_threshold": "5", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_tick_period": "2", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_accept_timeout_factor": "2.000000", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_allow_pool_delete": "true", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_allow_pool_size_one": "true", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_auth_validate_all_caps": "true", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_bytes": "104857600", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_directed_command_retry": "5", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_hunt_interval": "3.000000", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-12-06T09:22:15.303 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_hunt_parallel": "3", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_log_interval": "1.000000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_ping_interval": "10.000000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_ping_timeout": "30.000000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_client_target_rank": "-1", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_clock_drift_allowed": "1.000000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_file_level": "debug", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_file": "true", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_graylog": "false", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_journald": "false", 2023-12-06T09:22:15.304 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_stderr": "false", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_compact_on_bootstrap": "false", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_compact_on_start": "false", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_compact_on_trim": "true", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_con_tracker_persist_interval": "10", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_con_tracker_score_halflife": "43200", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_config_key_max_entry_size": "65536", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_cpu_threads": "4", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_crush_min_required_version": "hammer", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_daemon_bytes": "419430400", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_data_avail_crit": "5", 2023-12-06T09:22:15.305 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_data_avail_warn": "5", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_data_size_warn": "16106127360", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_block_osdmap_trim": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_dump_json": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_dump_transactions": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_extra_checks": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_no_require_quincy": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_no_require_reef": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_delta_reset_interval": "10.000000", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_dns_srv_name": "ceph-mon", 2023-12-06T09:22:15.306 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_down_mkfs_grace": "120", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_election_default_strategy": "3", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_election_timeout": "5.000000", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_elector_ping_divisor": "2", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_elector_ping_timeout": "2.000000", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_enable_op_tracker": "true", 2023-12-06T09:22:15.307 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_fake_pool_delete": "false", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_force_quorum_join": "false", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_globalid_prealloc": "10000", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_health_detail_to_clog": "false", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_health_log_update_period": "5", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_health_max_detail": "50", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_health_to_clog": "true", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_health_to_clog_interval": "600", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_host": "172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],172.21.15.132", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_host_override": "", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_initial_members": "", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-12-06T09:22:15.308 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_keyvaluedb": "rocksdb", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_lease": "5.000000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_log_full_interval": "50", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_log_max": "10000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_log_max_summary": "50", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_log_entries_per_event": "4096", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_log_epochs": "500", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_mdsmap_epochs": "500", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_mgrmap_epochs": "500", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_osd": "10000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_pg_per_osd": "10000", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_pool_pg_num": "65536", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-12-06T09:22:15.309 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mds_force_trim_to": "0", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mds_skip_sanity": "false", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_memory_autotune": "true", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_memory_target": "2147483648", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mgr_beacon_grace": "30", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mgr_digest_period": "5", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mgr_inactive_grace": "60", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mgr_mkfs_grace": "240", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_min_osdmap_epochs": "50", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_op_complaint_time": "30", 2023-12-06T09:22:15.310 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_op_history_duration": "600", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_op_history_size": "20", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_op_history_slow_op_size": "20", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_op_history_slow_op_threshold": "10", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_op_log_threshold": "5", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_auto_mark_in": "false", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_auto_mark_new_in": "true", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_cache_size": "500", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_cache_size_min": "134217728", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_crush_smoke_test": "false", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_destroyed_out_interval": "600", 2023-12-06T09:22:15.311 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_down_out_interval": "600", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_force_trim_to": "0", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_full_ratio": "0.950000", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_laggy_halflife": "3600", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_laggy_max_interval": "300", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_laggy_weight": "0.300000", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_max_creating_pgs": "1024", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_max_initial_pgs": "1024", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_min_down_reporters": "2", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-12-06T09:22:15.312 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_prime_pg_temp": "true", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_report_timeout": "900", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_warn_num_repaired": "10", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osd_warn_op_age": "32.000000", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osdmap_full_prune_interval": "2", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osdmap_full_prune_min": "15", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_osdmap_full_prune_txsize": "2", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pg_stuck_threshold": "60", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-12-06T09:22:15.313 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pg_warn_min_objects": "10000", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pg_warn_min_per_osd": "0", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pool_quota_crit_threshold": "0", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_pool_quota_warn_threshold": "0", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_probe_timeout": "2.000000", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_reweight_max_change": "0.050000", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_reweight_max_osds": "4", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_scrub_interval": "300", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_scrub_max_keys": "100", 2023-12-06T09:22:15.314 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_scrub_timeout": "300", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_session_timeout": "300", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_smart_report_timeout": "5", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_stat_smooth_intervals": "6", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_stretch_pool_min_size": "2", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_stretch_pool_size": "4", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_subscribe_interval": "86400.000000", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_sync_debug": "false", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_sync_max_payload_keys": "2000", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_sync_max_payload_size": "1048576", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_sync_provider_kill_at": "0", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_sync_requester_kill_at": "0", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_sync_timeout": "60.000000", 2023-12-06T09:22:15.315 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_target_pg_per_osd": "100", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_tick_interval": "5", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_timecheck_interval": "300.000000", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_use_min_delay_socket": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_older_version_delay": "604800", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_filestore_osds": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_misplaced": "false", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-12-06T09:22:15.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_older_version": "true", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_pool_no_app": "true", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_on_too_few_osds": "false", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "monmap": "", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-12-06T09:22:15.317 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "motr_tracing_enabled": "false", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_op_threads": "3", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_buffer_size": "131072", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_cm": "false", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_device_name": "", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_dscp": "96", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_gid_idx": "0", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_local_gid": "", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_polling_us": "1000", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_port_num": "1", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-12-06T09:22:15.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_roce_ver": "1", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_send_buffers": "1024", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_sl": "3", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_support_srq": "true", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_rdma_type": "ib", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_async_reap_threshold": "5", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_before_connect": "false", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_ipv4": "true", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_ipv6": "false", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_msgr1": "false", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_msgr2": "true", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_port_max": "7568", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_port_min": "6800", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_prefer_ipv4": "false", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_retry_count": "3", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_bind_retry_delay": "5", 2023-12-06T09:22:15.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_blackhole_client": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_blackhole_mds": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_blackhole_mgr": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_blackhole_mon": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_blackhole_osd": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_client_mode": "crc secure", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_cluster_mode": "crc secure", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_cluster_type": "", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_compress_secure": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_connection_idle_timeout": "900", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_connection_ready_timeout": "10", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_crc_data": "true", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_crc_header": "true", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_die_on_bad_msg": "false", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_die_on_bug": "true", 2023-12-06T09:22:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_die_on_old_message": "true", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_die_on_skipped_message": "false", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_die_on_unhandled_msg": "false", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_coremask": "0xF", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_devs_allowlist": "", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_enable_tso": "true", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_hugepages": "", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_hw_flow_control": "true", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_lro": "true", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_memory_channel": "4", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-12-06T09:22:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_pmd": "", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_port_id": "0", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dump_corrupt_message_level": "1", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_dump_on_send": "false", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_initial_backoff": "0.200000", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_inject_delay_max": "1.000000", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_inject_delay_probability": "0.000000", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_inject_delay_type": "", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_inject_internal_delays": "0.000000", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_inject_network_congestion": "0", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_inject_socket_failures": "5000", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_learn_addr_from_peer": "true", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_max_accept_failures": "4", 2023-12-06T09:22:15.322 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_max_backoff": "15.000000", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_mon_client_mode": "secure crc", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_mon_cluster_mode": "secure crc", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_mon_service_mode": "secure crc", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_osd_compress_min_size": "1024", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_osd_compress_mode": "none", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_osd_compression_algorithm": "snappy", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_pq_min_cost": "65536", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_public_type": "", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_service_mode": "crc secure", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_tcp_listen_backlog": "512", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_tcp_nodelay": "true", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_tcp_rcvbuf": "0", 2023-12-06T09:22:15.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ms_type": "async", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "no_config_file": "false", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_completion_locks_per_session": "32", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_debug_inject_relock_delay": "false", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_inflight_op_bytes": "104857600", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_inflight_ops": "1024", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_inject_no_watch_ping": "false", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_tick_interval": "5.000000", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter_timeout": "10.000000", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "objectstore_blackhole": "false", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "openssl_engine_opts": "", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_delay_time": "5.000000", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_hist_halflife": "1000", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_max_low_ops": "2", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_max_ops": "4", 2023-12-06T09:22:15.324 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_quantize_effort": "0.100000", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_agent_slop": "0.020000", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_allow_recovery_below_min_size": "true", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_asio_thread_count": "2", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_async_recovery_min_cost": "100", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_auto_mark_unfound_lost": "false", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_backfill_retry_interval": "30.000000", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_backfill_scan_max": "512", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_backfill_scan_min": "64", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_backoff_on_degraded": "false", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_backoff_on_peering": "false", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_backoff_on_unfound": "true", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_beacon_report_interval": "300", 2023-12-06T09:22:15.325 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bench_duration": "30", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bench_max_block_size": "67108864", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bench_small_size_max_iops": "100", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_blkin_trace_all": "false", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_blocked_scrub_grace_period": "120", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_check_for_log_corruption": "false", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_class_update_on_start": "true", 2023-12-06T09:22:15.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_client_message_cap": "256", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_client_message_size_cap": "524288000", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_client_op_priority": "63", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_client_watch_timeout": "30", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_command_max_records": "256", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_compact_on_start": "false", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_crush_chooseleaf_type": "0", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_crush_initial_weight": "-1.000000", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_crush_update_on_start": "true", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_crush_update_weight_set": "true", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_drop_ping_duration": "0", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-12-06T09:22:15.327 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_feed_pullee": "-1", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_misdirected_ops": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_no_acting_change": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_no_purge_strays": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_op_order": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_pg_log_writeout": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_pretend_recovery_active": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_shutdown": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-12-06T09:22:15.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_trim_objects": "false", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_verify_cached_snaps": "false", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_verify_missing_on_start": "false", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_verify_snaps": "false", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_keys": "1024", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_stride": "524288", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_default_data_pool_replay_window": "5", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_default_notify_timeout": "30", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_delete_sleep": "0.000000", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-12-06T09:22:15.329 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_discard_disconnected_ops": "true", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_enable_op_tracker": "true", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_extblkdev_plugins": "vdo", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_fast_info": "true", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_fast_shutdown": "true", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_fast_shutdown_timeout": "15", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-12-06T09:22:15.330 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_function_tracing": "false", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_grace": "80", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_interval": "6", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_min_peers": "10", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_min_size": "2000", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_stale": "600", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_hit_set_max_size": "100000", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_hit_set_min_size": "1000", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-12-06T09:22:15.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_journal_flush_on_shutdown": "true", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_journal_size": "5120", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_kill_backfill_at": "0", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_loop_before_reset_tphandle": "64", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_map_cache_size": "50", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_map_dedup": "true", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_map_message_max": "40", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_map_message_max_bytes": "10485760", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_map_share_max_epochs": "40", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_attr_name_len": "100", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_attr_size": "0", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_backfills": "1", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_markdown_count": "5", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_markdown_period": "600", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_object_name_len": "2048", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_object_namespace_len": "256", 2023-12-06T09:22:15.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_object_size": "134217728", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_omap_entries_per_request": "1024", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_pg_blocked_by": "16", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_pg_log_entries": "10000", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_pgls": "1024", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_push_cost": "8388608", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_push_objects": "10", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_scrubs": "3", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_trimming_pgs": "2", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_write_op_reply_len": "64", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_max_write_size": "90", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-12-06T09:22:15.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_override_recovery_settings": "false", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_profile": "balanced", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-12-06T09:22:15.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mclock_skip_benchmark": "false", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_base": "805306368", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_cache_min": "134217728", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_target": "4294967296", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_target_autotune": "false", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_min_pg_log_entries": "250", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_min_recovery_priority": "0", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mon_heartbeat_interval": "30", 2023-12-06T09:22:15.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mon_report_interval": "5", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mon_report_max_in_flight": "2", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_num_cache_shards": "32", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_num_op_tracker_shard": "32", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_numa_auto_affinity": "true", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_numa_node": "-1", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_numa_prefer_iface": "true", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_objecter_finishers": "1", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_objectstore": "bluestore", 2023-12-06T09:22:15.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_objectstore_fuse": "false", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_objectstore_tracing": "false", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_complaint_time": "30.000000", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_history_duration": "600", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_history_size": "20", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_history_slow_op_size": "20", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_log_threshold": "5", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_num_shards": "0", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_num_shards_hdd": "5", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_num_shards_ssd": "8", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_num_threads_per_shard": "0", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-12-06T09:22:15.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_pq_min_cost": "65536", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_queue": "mclock_scheduler", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_queue_cut_off": "high", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_thread_suicide_timeout": "150", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_op_thread_timeout": "15", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_open_classes_on_start": "true", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_os_flags": "0", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_peering_op_priority": "255", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_delete_cost": "1048576", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_delete_priority": "5", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_log_dups_tracked": "3000", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_log_trim_max": "10000", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_log_trim_min": "100", 2023-12-06T09:22:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_object_context_cache_count": "64", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pg_stat_report_interval_max": "500", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_crimson": "false", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_crush_rule": "-1", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_ec_fast_read": "false", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_flag_bulk": "false", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_flag_nodelete": "false", 2023-12-06T09:22:15.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_flags": "0", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_min_size": "0", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_pg_num": "32", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_pgp_num": "0", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_size": "2", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_default_type": "replicated", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pool_use_gmt_hitset": "true", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_push_per_object_cost": "1000", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_read_ec_check_for_errors": "false", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recover_clone_overlap": "true", 2023-12-06T09:22:15.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recover_clone_overlap_limit": "10", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_cost": "20971520", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_delay_start": "0.000000", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_max_active": "0", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_max_active_hdd": "3", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_max_active_ssd": "10", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_max_chunk": "8388608", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_max_single_start": "1", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_op_priority": "3", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_op_warn_multiple": "16", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_priority": "5", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_retry_interval": "30.000000", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_sleep": "0.000000", 2023-12-06T09:22:15.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_repair_during_recovery": "false", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_requested_scrub_priority": "120", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_rollback_to_cluster_snap": "", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_auto_repair": "false", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_begin_hour": "0", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_begin_week_day": "0", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_chunk_max": "25", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_chunk_min": "5", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_cost": "52428800", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_during_recovery": "false", 2023-12-06T09:22:15.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_end_hour": "0", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_end_week_day": "0", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_event_cost": "4096", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_invalid_stats": "true", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_load_threshold": "0.500000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_max_interval": "604800.000000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_max_preemptions": "5", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_min_interval": "86400.000000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_priority": "5", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_reservation_timeout": "300000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_sleep": "0.000000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_scrub_slow_reservation_response": "30000", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-12-06T09:22:15.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_shutdown_pgref_assert": "false", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_skip_data_digest": "false", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_smart_report_timeout": "5", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_snap_trim_cost": "1048576", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_snap_trim_priority": "5", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_snap_trim_sleep": "0.000000", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-12-06T09:22:15.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_stats_update_period_scrubbing": "15", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_target_transaction_size": "30", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_default_cache_mode": "writeback", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_tracing": "false", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_use_stale_snap": "false", 2023-12-06T09:22:15.345 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "osdc_blkin_trace_all": "false", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_kill_at": "0", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_max_join_drift": "10", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_min": "500", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_min_wait": "0.050000", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_propose_interval": "1.000000", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_service_trim_max": "500", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_service_trim_max_multiplier": "20", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_service_trim_min": "10", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_stash_full_interval": "25", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_trim_max": "500", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "paxos_trim_min": "250", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "perf": "true", 2023-12-06T09:22:15.346 INFO:teuthology.orchestra.run.smithi088.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin_dir": "/usr/lib64/ceph", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "-", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrv": "", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "public_bind_addr": "-", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "public_network": "", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "public_network_interface": "", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "qat_compressor_enabled": "false", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "qat_compressor_session_max_number": "256", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "rados_mon_op_timeout": "0", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "rados_osd_op_timeout": "0", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "rados_tracing": "false", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_atime_update_interval": "60", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-12-06T09:22:15.347 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_balance_parent_reads": "false", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_balance_snap_reads": "false", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_blkin_trace_all": "false", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_blocklist_expire_seconds": "0", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_blocklist_on_break_lock": "true", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache": "true", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_block_writes_upfront": "false", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_max_dirty": "25165824", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_max_dirty_object": "0", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_policy": "writearound", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_size": "33554432", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_target_dirty": "16777216", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_clone_copy_on_read": "false", 2023-12-06T09:22:15.348 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_compression_hint": "none", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_concurrent_management_ops": "10", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_clone_format": "auto", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_data_pool": "", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_features": "61", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_format": "2", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_map_options": "", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_order": "22", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_pool": "rbd", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_stripe_count": "0", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_default_stripe_unit": "0", 2023-12-06T09:22:15.349 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_disable_zero_copy_writes": "true", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_discard_granularity_bytes": "65536", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_enable_alloc_hint": "true", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_io_scheduler": "simple", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_commit_age": "5.000000", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_object_flush_interval": "0", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-12-06T09:22:15.350 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_order": "24", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_pool": "", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_journal_splay_width": "4", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_localize_parent_reads": "false", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_localize_snap_reads": "false", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_image_policy_type": "simple", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-12-06T09:22:15.351 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_autotune": "true", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_base": "805306368", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_memory_target": "4294967296", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirroring_delete_delay": "0", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-12-06T09:22:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirroring_replay_delay": "0", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_move_to_trash_on_remove": "false", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_mtime_update_interval": "60", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_non_blocking_aio": "true", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_op_thread_timeout": "60", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_op_threads": "1", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_parent_cache_enabled": "false", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_persistent_cache_mode": "disabled", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_persistent_cache_path": "/tmp", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_persistent_cache_size": "1073741824", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_plugins": "", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_bps_burst": "0", 2023-12-06T09:22:15.353 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_bps_limit": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_exclude_ops": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_iops_burst": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_iops_limit": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_read_bps_burst": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_read_bps_limit": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_read_iops_burst": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_read_iops_limit": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_schedule_tick_min": "50", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_write_bps_burst": "0", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-12-06T09:22:15.354 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_write_bps_limit": "0", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_write_iops_burst": "0", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_qos_write_iops_limit": "0", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_quiesce_notification_attempts": "10", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_read_from_replica_policy": "default", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_readahead_max_bytes": "524288", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_readahead_trigger_requests": "10", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_request_timed_out_seconds": "30", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_skip_partial_discard": "true", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_tracing": "false", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_validate_names": "true", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd_validate_pool": "true", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "restapi_base_url": "", 2023-12-06T09:22:15.355 INFO:teuthology.orchestra.run.smithi088.stdout: "restapi_log_level": "", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_acl_grants_max_num": "100", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_admin_entry": "admin", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_backend_store": "rados", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_barbican_url": "", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_beast_enable_async": "true", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_counters_cache": "false", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_counters_cache_size": "10000", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_index_max_aio": "128", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_quota_ttl": "600", 2023-12-06T09:22:15.356 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_cache_enabled": "true", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_cache_expiry_interval": "900", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_cache_lru_size": "10000", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_config_store": "rados", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_content_length_compat": "false", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_copy_obj_progress": "true", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_cors_rules_max_num": "100", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_cross_domain_policy": "", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_default_encryption_key": "", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_addr": "", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_ca_path": "", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_client_cert": "", 2023-12-06T09:22:15.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_client_key": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_password": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_kmip_username": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_require_ssl": "true", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-12-06T09:22:15.358 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_suppress_logs": "true", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_addr": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_auth": "token", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_namespace": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_prefix": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_token_file": "", 2023-12-06T09:22:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_curl_buffersize": "524288", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_curl_low_speed_limit": "1024", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_curl_low_speed_time": "300", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_curl_tcp_keepalive": "0", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_l1_fadvise": "4", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d4n_host": "127.0.0.1", 2023-12-06T09:22:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_d4n_port": "6379", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_log_changes_size": "1000", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_log_num_shards": "128", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_log_window": "30", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_notify_interval_msec": "0", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_sync_poll_interval": "20", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_data_sync_spawn_window": "20", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_default_data_log_backing": "fifo", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_default_region_info_oid": "default.region", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-12-06T09:22:15.361 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_defer_to_bucket_acls": "", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_data_res": "500.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dns_name": "", 2023-12-06T09:22:15.362 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dns_s3website_name": "", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_dynamic_resharding": "true", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_gc_threads": "true", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_lc_threads": "true", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_ops_log": "false", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_quota_threads": "true", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_static_website": "false", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enable_usage_log": "false", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_enforce_swift_acls": "true", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_exit_timeout_secs": "120", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_expose_bucket": "false", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_extended_http_attrs": "", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_filter": "none", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_frontends": "beast port=7480", 2023-12-06T09:22:15.363 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_max_concurrent_io": "10", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_max_deferred": "50", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_max_objs": "32", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_max_queue_size": "134213632", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_max_trim_chunk": "16", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_obj_min_wait": "7200", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_processor_max_time": "3600", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_gc_processor_period": "3600", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_get_obj_window_size": "16777216", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_healthcheck_disabling_path": "", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ignore_get_invalid_range": "false", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_init_timeout": "300", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-12-06T09:22:15.364 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_domain": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_password": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_password_path": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_project": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_tenant": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_token": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_token_path": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_admin_user": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_api_version": "2", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_barbican_domain": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_barbican_password": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_barbican_project": "", 2023-12-06T09:22:15.365 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_barbican_tenant": "", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_barbican_user": "", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_implicit_tenants": "false", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_service_token_enabled": "false", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_token_cache_size": "10000", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_url": "", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_keystone_verify_ssl": "true", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_debug_interval": "-1", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_lock_max_time": "90", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_max_objs": "32", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_max_rules": "1000", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_max_worker": "3", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_max_wp_worker": "3", 2023-12-06T09:22:15.366 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lc_thread_delay": "0", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ldap_dnattr": "uid", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ldap_searchfilter": "", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ldap_uri": "ldaps://", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_log_http_headers": "", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_log_nonexistent_bucket": "false", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_log_object_name_utc": "false", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-12-06T09:22:15.367 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_attr_name_len": "0", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_attr_size": "0", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_attrs_num_in_req": "0", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_chunk_size": "4194304", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_concurrent_requests": "1024", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_dynamic_shards": "1999", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_listing_results": "1000", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_notify_retries": "10", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_objs_per_shard": "100000", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_put_param_size": "1048576", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_put_size": "5368709120", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_max_slo_entries": "1000", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_md_log_max_shards": "64", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_md_notify_interval_msec": "200", 2023-12-06T09:22:15.368 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_meta_sync_poll_interval": "20", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_meta_sync_spawn_window": "20", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_mp_lock_max_time": "600", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_multipart_min_part_size": "5242880", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_fhcache_size": "2017", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_lru_lanes": "5", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_max_gc": "300", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_run_gc_threads": "false", 2023-12-06T09:22:15.369 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_run_lc_threads": "false", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_run_quota_threads": "false", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_run_sync_thread": "false", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_num_async_rados_threads": "32", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_num_control_oids": "8", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_numa_node": "-1", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_obj_stripe_size": "4194304", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_objexp_chunk_size": "100", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_objexp_gc_interval": "600", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_objexp_hints_num_shards": "127", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-12-06T09:22:15.370 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_op_thread_timeout": "600", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_op_tracing": "false", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_opa_token": "", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_opa_url": "", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_opa_verify_ssl": "true", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ops_log_rados": "false", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_ops_log_socket_path": "", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_period_push_interval": "2.000000", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_period_push_interval_max": "30.000000", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_period_root_pool": ".rgw.root", 2023-12-06T09:22:15.371 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_posix_cache_lanes": "3", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_posix_cache_max_buckets": "100", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_posix_cache_partitions": "3", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_print_continue": "true", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_print_prohibited_content_length": "false", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_rados_tracing": "false", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_realm": "", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_realm_id": "", 2023-12-06T09:22:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_region": "", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_region_root_pool": ".rgw.root", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_relaxed_region_enforcement": "false", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_request_uri": "", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_reshard_batch_size": "64", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_reshard_max_aio": "128", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_reshard_num_logs": "16", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_reshard_thread_interval": "600", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_resolve_cname": "false", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_rest_getusage_op_compat": "false", 2023-12-06T09:22:15.373 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_run_sync_thread": "true", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_auth_use_keystone": "false", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_auth_use_ldap": "false", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_auth_use_rados": "true", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_auth_use_sts": "false", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_s3_success_create_obj_status": "0", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_scheduler_type": "throttler", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_script_uri": "", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_service_provider_name": "", 2023-12-06T09:22:15.374 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_client_id": "", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_client_secret": "", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_entry": "sts", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_key": "sts", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_max_session_duration": "43200", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_min_session_duration": "900", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sts_token_introspection_url": "", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_account_in_url": "false", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_auth_entry": "auth", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_auth_url": "", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_custom_header": "", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_enforce_content_length": "false", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_need_stats": "true", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_tenant_name": "", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_token_expiration": "86400", 2023-12-06T09:22:15.375 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_url": "", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_url_prefix": "swift", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_swift_versioning_enabled": "false", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_lease_period": "120", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_log_trim_interval": "1200", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_obj_etag_verify": "false", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_trace_history_size": "4096", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_thread_pool_size": "512", 2023-12-06T09:22:15.376 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_topic_persistency_max_retries": "0", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_comment": "", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_createby": "", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_encoding": "", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_flag": "false", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_max_size": "5368709120", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_origin": "", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_sha_unit": "524288", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_torrent_tracker": "", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_trust_forwarded_https": "false", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_usage_log_tick_interval": "30", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_usage_max_shards": "32", 2023-12-06T09:22:15.377 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_usage_max_user_shards": "1", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_use_opa_authz": "false", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_counters_cache": "false", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_counters_cache_size": "10000", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_default_quota_max_size": "-1", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_max_buckets": "1000", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_policies_max_num": "100", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_quota_sync_interval": "86400", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_user_unique_email": "true", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_verify_ssl": "true", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_website_routing_rules_max_num": "50", 2023-12-06T09:22:15.378 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_zone": "", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_zone_id": "", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_zonegroup": "", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_zonegroup_id": "", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgwlc_auto_session_clear": "true", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rgwlc_skip_bucket_step": "false", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_block_size": "4096", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cache_shard_bits": "4", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cache_size": "536870912", 2023-12-06T09:22:15.379 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cache_type": "binned_lru", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_collect_compaction_stats": "false", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_collect_extended_stats": "false", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_collect_memory_stats": "false", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_index_type": "binary_search", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_log_to_ceph_log": "true", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_metadata_block_size": "4096", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_partition_filters": "false", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_perf": "false", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "rotating_keys_renewal_timeout": "10", 2023-12-06T09:22:15.380 INFO:teuthology.orchestra.run.smithi088.stdout: "run_dir": "/var/run/ceph", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_block_create": "true", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_cache_lru_size": "67108864", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_cbjournal_size": "5368709120", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_data_delta_based_overwrite": "0", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_default_max_object_size": "16777216", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_device_size": "53687091200", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_journal_batch_capacity": "16", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_journal_iodepth_limit": "5", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_main_device_type": "SSD", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_max_concurrent_transactions": "8", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-12-06T09:22:15.381 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "seastore_segment_size": "67108864", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "set_keepcaps": "false", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "setgroup": "", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "setuser": "", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "setuser_match_path": "", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_misplaced_ratio": "0.050000", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "thp": "false", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "threadpool_default_timeout": "60", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "threadpool_empty_queue_max_wait": "2", 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout: "throttler_perf_counter": "true" 2023-12-06T09:22:15.382 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:15.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-12-06T09:22:15.635 INFO:teuthology.orchestra.run.smithi088.stdout:mon.a: {} 2023-12-06T09:22:15.636 INFO:teuthology.orchestra.run.smithi088.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T09:22:15.731 INFO:teuthology.orchestra.run.smithi088.stdout:mon.b: {} 2023-12-06T09:22:15.731 INFO:teuthology.orchestra.run.smithi088.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T09:22:15.826 INFO:teuthology.orchestra.run.smithi088.stdout:mon.c: {} 2023-12-06T09:22:15.827 INFO:teuthology.orchestra.run.smithi088.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T09:22:15.838 DEBUG:teuthology.orchestra.run.smithi088:> type ceph-objectstore-tool 2023-12-06T09:22:15.852 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-06T09:22:15.852 DEBUG:teuthology.orchestra.run.smithi132:> type ceph-objectstore-tool 2023-12-06T09:22:15.867 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-06T09:22:15.867 INFO:teuthology.run_tasks:Running task mon_recovery... 2023-12-06T09:22:15.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:22:15.878 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-12-06T09:22:15.878 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:22:15.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:22:15.878 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-12-06T09:22:15.879 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-12-06T09:22:15.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:22:15.880 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-12-06T09:22:15.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:15.880 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-12-06T09:22:15.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:22:15.914 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:15.926+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:15.986 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:15.998+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:16.058 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:16.058 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:16.058 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:16.059 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:16.075 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:16.082 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:16.096+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:16.190 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:16.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:16.228 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:16.229 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:16.269 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:16.269 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:16.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:16.283 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:16.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:16.389 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:16.402+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:16.394 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:16.395 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:16.491 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:16.503+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:16.502 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:16.503 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:16.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:16.537 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:16.538 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:16.594 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:16.604+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:16.688 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:16.702+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:16.692 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:16.711 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:16.712 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:22:04.327786+0000","last_up_change":"2023-12-06T09:21:55.261571+0000","last_in_change":"2023-12-06T09:21:13.597313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T09:22:01.832571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4165054852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":4165054852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":4165054852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4165054852}]},"public_addr":"172.21.15.132:6812/4165054852","cluster_addr":"172.21.15.132:6813/4165054852","heartbeat_back_addr":"172.21.15.132:6815/4165054852","heartbeat_front_addr":"172.21.15.132:6814/4165054852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:22:16.725 INFO:tasks.mon_recovery.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T09:21:54.386974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T09:22:01.832571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-06T09:22:16.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-06T09:22:16.731 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:16.732 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:16.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:16.813 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:16.813 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:16.814 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:16.827+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:16.892 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:16.905+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:16.918 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:16.919 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:16.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:16.941 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:16.941 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:16.992 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:17.007+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:17.066 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:17.066 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:17.072 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:17.072 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:17.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:17.102 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:17.114+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:17.188 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:17.201 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:17.215+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:17.262 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:17.262 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:17.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:17.304 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:17.318+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:17.326 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-12-06T09:22:17.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-12-06T09:22:17.410 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:17.422+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:17.411 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:17.411 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:17.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:17.512 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:17.522+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:17.571 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:17.571 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:17.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:17.609 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:17.624+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:17.712 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:17.727+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:17.754 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:17.754 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:17.767 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:22:17.824 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:17.836+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:17.916 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:17.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:17.957 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 8 2023-12-06T09:22:17.965 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:17.965 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:17.972 INFO:tasks.mon_recovery:mon ids = ['a', 'c', 'b'] 2023-12-06T09:22:17.972 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:22:17.972 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:17.975 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:18.017 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:18.031+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:18.120 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:18.134+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:18.154 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:18.154 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:18.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:18.221 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:18.235+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:18.299 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:18.299 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:18.308 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:18.323 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:18.338+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:18.424 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:18.438+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:18.438 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:18.439 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:18.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:18.526 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:18.540+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:18.617 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:18.617 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:18.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:18.627 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:18.641+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:18.671 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":70,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:18.685 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:22:18.685 INFO:tasks.mon_recovery:verifying all monitors are in the quorum 2023-12-06T09:22:18.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-06T09:22:18.729 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:18.743+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:18.765 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:18.765 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:18.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:18.829 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:18.844+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:18.906 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:18.906 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:18.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:18.932 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:18.932 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "a", 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 0, 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "leader", 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "election_epoch": 10, 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum": [ 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: 0, 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: 1, 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: 2 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_age": 70, 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "features": { 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "required_con": "2449958758054445060", 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "required_mon": [ 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:18.933 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_con": "4540138322906710015", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_mon": [ 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:18.934 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "outside_quorum": [], 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "extra_probe_peers": [], 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "sync_provider": [], 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "monmap": { 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 1, 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:20:12.041363Z", 2023-12-06T09:22:18.935 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:20:12.041363Z", 2023-12-06T09:22:18.936 INFO:teuthology.orchestra.run.smithi088.stdout: "min_mon_release": 18, 2023-12-06T09:22:18.937 INFO:teuthology.orchestra.run.smithi088.stdout: "min_mon_release_name": "reef", 2023-12-06T09:22:18.937 INFO:teuthology.orchestra.run.smithi088.stdout: "election_strategy": 3, 2023-12-06T09:22:18.937 INFO:teuthology.orchestra.run.smithi088.stdout: "disallowed_leaders: ": "", 2023-12-06T09:22:18.937 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": false, 2023-12-06T09:22:18.938 INFO:teuthology.orchestra.run.smithi088.stdout: "tiebreaker_mon": "", 2023-12-06T09:22:18.938 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_ranks: ": "", 2023-12-06T09:22:18.938 INFO:teuthology.orchestra.run.smithi088.stdout: "features": { 2023-12-06T09:22:18.938 INFO:teuthology.orchestra.run.smithi088.stdout: "persistent": [ 2023-12-06T09:22:18.945 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:18.945 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:18.945 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:18.945 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:18.945 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:18.945 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "optional": [] 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "mons": [ 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 0, 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "a", 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:18.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:3300", 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6789", 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6789/0", 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6789/0", 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:18.947 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 1, 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "b", 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:3300", 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6789", 2023-12-06T09:22:18.948 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6789/0", 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6789/0", 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 2, 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "c", 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:18.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:3301", 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6790", 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6790/0", 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6790/0", 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:18.950 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "feature_map": { 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": [ 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 1 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": [ 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:18.951 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 2 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "client": [ 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 3 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": [ 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:18.952 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 2 2023-12-06T09:22:18.953 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:18.953 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:18.953 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:18.953 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": false 2023-12-06T09:22:18.953 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:18.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-06T09:22:18.959 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:18.973+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:19.039 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:19.053+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:19.042 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:19.042 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:19.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:19.135 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:19.149+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:19.191 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "c", 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 2, 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "peon", 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "election_epoch": 10, 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum": [ 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: 0, 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: 1, 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: 2 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_age": 71, 2023-12-06T09:22:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: "features": { 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "required_con": "2449958758054445060", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "required_mon": [ 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:19.193 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_con": "4540138322906710015", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_mon": [ 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:19.194 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:19.195 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: "outside_quorum": [], 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: "extra_probe_peers": [], 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: "sync_provider": [], 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: "monmap": { 2023-12-06T09:22:19.200 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 1, 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:20:12.041363Z", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:20:12.041363Z", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "min_mon_release": 18, 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "min_mon_release_name": "reef", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "election_strategy": 3, 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "disallowed_leaders: ": "", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": false, 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "tiebreaker_mon": "", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_ranks: ": "", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "features": { 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "persistent": [ 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "optional": [] 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "mons": [ 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 0, 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "a", 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:19.202 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:3300", 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6789", 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6789/0", 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6789/0", 2023-12-06T09:22:19.203 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 1, 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "b", 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:3300", 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.204 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6789", 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6789/0", 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6789/0", 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 2, 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "c", 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:19.205 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:19.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.206 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:19.206 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:3301", 2023-12-06T09:22:19.206 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6790", 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6790/0", 2023-12-06T09:22:19.208 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6790/0", 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "feature_map": { 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": [ 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 1 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.209 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": [ 2023-12-06T09:22:19.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.210 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:19.210 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:19.210 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 1 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: "client": [ 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 4 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": false 2023-12-06T09:22:19.211 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:19.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:22:19.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-06T09:22:19.236 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:19.250+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:19.237 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:19.237 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:19.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:19.339 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:19.353+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:19.424 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:19.425 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:19.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:19.449 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "b", 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 1, 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "peon", 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "election_epoch": 10, 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum": [ 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: 0, 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: 1, 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: 2 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_age": 71, 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "features": { 2023-12-06T09:22:19.450 INFO:teuthology.orchestra.run.smithi088.stdout: "required_con": "2449958758054445060", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "required_mon": [ 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.451 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_con": "4540138322906710015", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "quorum_mon": [ 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "outside_quorum": [], 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "extra_probe_peers": [], 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "sync_provider": [], 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "monmap": { 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 1, 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:20:12.041363Z", 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:20:12.041363Z", 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "min_mon_release": 18, 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "min_mon_release_name": "reef", 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "election_strategy": 3, 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "disallowed_leaders: ": "", 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": false, 2023-12-06T09:22:19.453 INFO:teuthology.orchestra.run.smithi088.stdout: "tiebreaker_mon": "", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_ranks: ": "", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "features": { 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "persistent": [ 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "kraken", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "luminous", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "mimic", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap-prune", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "nautilus", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "octopus", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "pacific", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "elector-pinging", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "quincy", 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: "reef" 2023-12-06T09:22:19.454 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "optional": [] 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "mons": [ 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 0, 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "a", 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:3300", 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6789", 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6789/0", 2023-12-06T09:22:19.456 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6789/0", 2023-12-06T09:22:19.457 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:19.457 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:19.457 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:19.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 1, 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "b", 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:3300", 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6789", 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6789/0", 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6789/0", 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "rank": 2, 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "name": "c", 2023-12-06T09:22:19.463 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:3301", 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6790", 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 0 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6790/0", 2023-12-06T09:22:19.464 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6790/0", 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "priority": 0, 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_location": "{}" 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "feature_map": { 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": [ 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 1 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.465 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": [ 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 5 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "client": [ 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "release": "luminous", 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 2 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": false 2023-12-06T09:22:19.466 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:19.468 INFO:tasks.mon_recovery:restarting each monitor in turn 2023-12-06T09:22:19.468 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T09:22:19.468 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:22:19.480 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:19.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:19.511 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:19.512 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:19.525 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:22:19.529+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86624) UID: 0 2023-12-06T09:22:19.526 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:22:19.531+0000 9f54640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T09:22:19.540 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:19.555+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:19.574 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:19.574 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:19.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:19.641 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:19.656+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:19.713 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:19.743 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:19.757+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:19.776 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:19.776 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:19.785 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:19.825 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:19.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:22:19.881 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:19.895+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:19.948 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:19.961+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:19.959 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:19.960 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:19.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:20.027 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:20.046 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:20.060+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:20.124 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:20.147 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:20.162+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:20.156 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:20.157 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:20.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:20.220 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:20.221 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:20.249 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:20.263+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:20.251 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:35.325 86645== 2023-12-06T09:22:20.251 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:35.325 86645== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:22:20.263 INFO:tasks.ceph.mon.a.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:22:20.319 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:20.320 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:20.320 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:20.320 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:20.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:20.351 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:20.364+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:20.451 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:20.465+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:20.463 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:20.463 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:20.463 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:20.463 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:20.463 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:20.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:20.557 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:20.571+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:20.643 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:20.695 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:20.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:20.696 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:20.696 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:20.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:20.698 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:20.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:20.799 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:20.813+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:20.870 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:20.871 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:20.871 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:20.871 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:20.879 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:22:20.879 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:22:20.880 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-12-06T09:22:20.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T09:22:20.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:20.899 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:20.914+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:21.001 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:21.015+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:21.055 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:21.055 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:21.055 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:21.055 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:21.055 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:21.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:21.109 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:21.123+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:21.202 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:21.216+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:21.262 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 24.342090629000001, 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:22:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:22:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:22:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:22:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.924380511999999, 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:22:21.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:22:21.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:22:21.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.284 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.920196066999999, 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:22:21.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:22:21.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:22:21.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.834664981, 2023-12-06T09:22:21.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:22:21.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:22:21.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:22:21.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:22:21.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:22:21.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.308 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.520688366000002, 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.519170522, 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:22:21.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:22:21.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:22:21.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.327 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.518505146999999, 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:22:21.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.514244975, 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:21.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:22:21.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.513611766, 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:22:21.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.512984664000001, 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.346 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:22:21.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:22:21.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.424724886, 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:22:21.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:22:21.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:22:21.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:22:21.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 21.423803906, 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:22:21.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:22:21.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:22:21.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 20.671425974999998, 2023-12-06T09:22:21.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:22:21.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:22:21.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:21.369 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.801391003999999, 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:22:21.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:22:21.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.800217493, 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:22:21.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:22:21.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.792065522, 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:22:21.381 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:22:21.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.387 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.781783586, 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:21.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:22:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:22:21.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.392 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.781125705999999, 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:22:21.393 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:22:21.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:22:21.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:22:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.780450927, 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:22:21.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:22:21.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:22:21.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:22:21.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:22:21.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 12.748093336, 2023-12-06T09:22:21.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:21.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:21.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:21.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:21.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:22:21.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:22:21.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:21.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.412 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:21.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:21.414 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:21.428+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:21.515 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:21.529+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:21.574 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:21.574 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:21.574 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 22, 2023-12-06T09:22:21.574 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:22:21.574 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:21:08.071156+0000", 2023-12-06T09:22:21.574 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:22:17.510276+0000", 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "last_up_change": "2023-12-06T09:21:55.261571+0000", 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "last_in_change": "2023-12-06T09:21:13.597313+0000", 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_num": 5801984, 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_set": [ 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "noscrub", 2023-12-06T09:22:21.575 INFO:teuthology.orchestra.run.smithi088.stdout: "pglog_hardlimit", 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "purged_snapdirs", 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_deletes", 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "sortbitwise" 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_version": 6, 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_snapshot": "", 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_max": 2, 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "max_osd": 8, 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "require_min_compat_client": "luminous", 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "min_compat_client": "jewel", 2023-12-06T09:22:21.576 INFO:teuthology.orchestra.run.smithi088.stdout: "require_osd_release": "reef", 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "allow_crimson": false, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "pools": [ 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 1, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": ".mgr", 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:21:54.386974+0000", 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:22:21.577 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 1, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 1, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 1, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 1, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 1, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:22:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "18", 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:22:21.579 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:22:21.580 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:22:21.581 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "options": { 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_max": 32, 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_min": 1 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.582 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": {} 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 8, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 8, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.25, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 2, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 2, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 1, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 1, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 1 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.583 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 2, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": "rbd", 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:22:01.832571+0000", 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 8, 2023-12-06T09:22:21.584 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 8, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 8, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 8, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 8, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.585 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "21", 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:22:21.586 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.587 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "options": {}, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd": {} 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.588 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 4.9800000190734863, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 4.9800000190734863, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 4.380000114440918, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 4.380000114440918, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 1, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 1, 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 1 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:21.589 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "db81b709-a318-4b43-beb2-bf3687f365e1", 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.590 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6805", 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.591 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6806", 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6808", 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6807", 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6805/2388339328", 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6806/2388339328", 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6808/2388339328", 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6807/2388339328", 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd7b0482-e237-424b-8eff-8db905713049", 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.594 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 15, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.595 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6801", 2023-12-06T09:22:21.597 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:21.597 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.597 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.597 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.597 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6802", 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6804", 2023-12-06T09:22:21.598 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6803", 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.600 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6801/3646708602", 2023-12-06T09:22:21.600 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6802/3646708602", 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6804/3646708602", 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6803/3646708602", 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e5a571be-6b8e-4c6f-bd3e-2af30c112285", 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.601 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6810", 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6813", 2023-12-06T09:22:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6816", 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6814", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6810/3714168299", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6813/3714168299", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6816/3714168299", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6814/3714168299", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.605 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "d7abe19a-cc3d-411f-8533-8592e1379735", 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.606 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6809", 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.607 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6811", 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6815", 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.608 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6812", 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6809/1828251745", 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6811/1828251745", 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1828251745", 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6812/1828251745", 2023-12-06T09:22:21.609 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b9517160-d081-44f9-aea4-6d2529e02114", 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 13, 2023-12-06T09:22:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6800", 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.612 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6801", 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.613 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6803", 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6802", 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.614 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6800/3720978101", 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6801/3720978101", 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6803/3720978101", 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6802/3720978101", 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e", 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:21.617 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:21.617 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.617 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.617 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.617 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6804", 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6805", 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6807", 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6806", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6804/1203998358", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6805/1203998358", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1203998358", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6806/1203998358", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e7afeca8-1363-4e9b-96f6-7fab6113d9e0", 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.621 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6808", 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6809", 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:21.623 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6811", 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.624 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6810", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6808/4151455937", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6809/4151455937", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6811/4151455937", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6810/4151455937", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd4b3971-3adb-44fd-b42b-b6660305beb3", 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:21.626 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.627 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.627 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6812", 2023-12-06T09:22:21.627 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:21.628 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.628 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6813", 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.629 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6815", 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6814", 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6812/4165054852", 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6813/4165054852", 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6815/4165054852", 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6814/4165054852", 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_xinfo": [ 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:22:21.631 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.229602+0000", 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.158659+0000", 2023-12-06T09:22:21.633 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.969728+0000", 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.637 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.234010+0000", 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.638 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:50.601730+0000", 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.639 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.304649+0000", 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.671399+0000", 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.055036+0000", 2023-12-06T09:22:21.641 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap": [], 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_items": [], 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_primaries": [], 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_temp": [], 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_temp": [], 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklist": {}, 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "range_blocklist": {}, 2023-12-06T09:22:21.642 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profiles": { 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "default": { 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "crush-failure-domain": "osd", 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "k": "2", 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "m": "1", 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin": "jerasure", 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "technique": "reed_sol_van" 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps_queue": [], 2023-12-06T09:22:21.643 INFO:teuthology.orchestra.run.smithi088.stdout: "new_removed_snaps": [], 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "new_purged_snaps": [], 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_node_flags": {}, 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "device_class_flags": {}, 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": { 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_enabled": false, 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_bucket_count": 0, 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "degraded_stretch_mode": 0, 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "recovering_stretch_mode": 0, 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_bucket": 0 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:21.646 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:21.646 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:21.646 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:21.646 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:21.647 INFO:tasks.thrashosds.thrasher:j is {'epoch': 22, 'fsid': '0c48cca3-dcc0-44bd-8eed-37fea2138550', 'created': '2023-12-06T09:21:08.071156+0000', 'modified': '2023-12-06T09:22:17.510276+0000', 'last_up_change': '2023-12-06T09:21:55.261571+0000', 'last_in_change': '2023-12-06T09:21:13.597313+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T09:21:54.386974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T09:22:01.832571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'db81b709-a318-4b43-beb2-bf3687f365e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6805', 'nonce': 2388339328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6806', 'nonce': 2388339328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6808', 'nonce': 2388339328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6807', 'nonce': 2388339328}]}, 'public_addr': '172.21.15.88:6805/2388339328', 'cluster_addr': '172.21.15.88:6806/2388339328', 'heartbeat_back_addr': '172.21.15.88:6808/2388339328', 'heartbeat_front_addr': '172.21.15.88:6807/2388339328', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'fd7b0482-e237-424b-8eff-8db905713049', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6801', 'nonce': 3646708602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6802', 'nonce': 3646708602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6804', 'nonce': 3646708602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6803', 'nonce': 3646708602}]}, 'public_addr': '172.21.15.88:6801/3646708602', 'cluster_addr': '172.21.15.88:6802/3646708602', 'heartbeat_back_addr': '172.21.15.88:6804/3646708602', 'heartbeat_front_addr': '172.21.15.88:6803/3646708602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e5a571be-6b8e-4c6f-bd3e-2af30c112285', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6810', 'nonce': 3714168299}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6813', 'nonce': 3714168299}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6816', 'nonce': 3714168299}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6814', 'nonce': 3714168299}]}, 'public_addr': '172.21.15.88:6810/3714168299', 'cluster_addr': '172.21.15.88:6813/3714168299', 'heartbeat_back_addr': '172.21.15.88:6816/3714168299', 'heartbeat_front_addr': '172.21.15.88:6814/3714168299', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd7abe19a-cc3d-411f-8533-8592e1379735', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6809', 'nonce': 1828251745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6811', 'nonce': 1828251745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6815', 'nonce': 1828251745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6812', 'nonce': 1828251745}]}, 'public_addr': '172.21.15.88:6809/1828251745', 'cluster_addr': '172.21.15.88:6811/1828251745', 'heartbeat_back_addr': '172.21.15.88:6815/1828251745', 'heartbeat_front_addr': '172.21.15.88:6812/1828251745', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b9517160-d081-44f9-aea4-6d2529e02114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 3720978101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6801', 'nonce': 3720978101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6803', 'nonce': 3720978101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 3720978101}]}, 'public_addr': '172.21.15.132:6800/3720978101', 'cluster_addr': '172.21.15.132:6801/3720978101', 'heartbeat_back_addr': '172.21.15.132:6803/3720978101', 'heartbeat_front_addr': '172.21.15.132:6802/3720978101', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1203998358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6805', 'nonce': 1203998358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6807', 'nonce': 1203998358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1203998358}]}, 'public_addr': '172.21.15.132:6804/1203998358', 'cluster_addr': '172.21.15.132:6805/1203998358', 'heartbeat_back_addr': '172.21.15.132:6807/1203998358', 'heartbeat_front_addr': '172.21.15.132:6806/1203998358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7afeca8-1363-4e9b-96f6-7fab6113d9e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 4151455937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6809', 'nonce': 4151455937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6811', 'nonce': 4151455937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 4151455937}]}, 'public_addr': '172.21.15.132:6808/4151455937', 'cluster_addr': '172.21.15.132:6809/4151455937', 'heartbeat_back_addr': '172.21.15.132:6811/4151455937', 'heartbeat_front_addr': '172.21.15.132:6810/4151455937', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fd4b3971-3adb-44fd-b42b-b6660305beb3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 4165054852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6813', 'nonce': 4165054852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6815', 'nonce': 4165054852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 4165054852}]}, 'public_addr': '172.21.15.132:6812/4165054852', 'cluster_addr': '172.21.15.132:6813/4165054852', 'heartbeat_back_addr': '172.21.15.132:6815/4165054852', 'heartbeat_front_addr': '172.21.15.132:6814/4165054852', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.229602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.158659+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.969728+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.234010+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:50.601730+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.304649+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.671399+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.055036+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T09:22:21.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:22:21.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:21.650 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:21.661+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:21.748 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:21.761+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:21.826 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:21.827 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:21.827 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:21.827 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:21.827 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:21.836 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:21.848 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:21.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:21.949 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:21.962+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:22.007 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:22.007 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:22.007 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:22.007 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:22.007 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:22.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:22.058 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:22.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:22.151 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:22.164+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:22.209 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:22.209 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:22.209 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:22.210 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:22.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:22.234 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:22:22.252 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:22:22.254 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:22.266+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:22.267 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-12-06T09:22:21.924380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217704,"kb_used_data":1544,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757016,"statfs":{"total":773094113280,"available":772871184384,"internally_reserved":0,"allocated":1581056,"data_stored":1319200,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001772"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.434185+0000","last_change":"2023-12-06T09:22:04.084324+0000","last_active":"2023-12-06T09:22:06.434185+0000","last_peered":"2023-12-06T09:22:06.434185+0000","last_clean":"2023-12-06T09:22:06.434185+0000","last_became_active":"2023-12-06T09:22:04.075169+0000","last_became_peered":"2023-12-06T09:22:04.075169+0000","last_unstale":"2023-12-06T09:22:06.434185+0000","last_undegraded":"2023-12-06T09:22:06.434185+0000","last_fullsized":"2023-12-06T09:22:06.434185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:06.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.778271+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:06.778271+0000","last_peered":"2023-12-06T09:22:06.778271+0000","last_clean":"2023-12-06T09:22:06.778271+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:06.778271+0000","last_undegraded":"2023-12-06T09:22:06.778271+0000","last_fullsized":"2023-12-06T09:22:06.778271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.433253+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:06.433253+0000","last_peered":"2023-12-06T09:22:06.433253+0000","last_clean":"2023-12-06T09:22:06.433253+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:06.433253+0000","last_undegraded":"2023-12-06T09:22:06.433253+0000","last_fullsized":"2023-12-06T09:22:06.433253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:04.520204+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:04.520204+0000","last_peered":"2023-12-06T09:22:04.520204+0000","last_clean":"2023-12-06T09:22:04.520204+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:04.520204+0000","last_undegraded":"2023-12-06T09:22:04.520204+0000","last_fullsized":"2023-12-06T09:22:04.520204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.430186+0000","last_change":"2023-12-06T09:22:04.156624+0000","last_active":"2023-12-06T09:22:06.430186+0000","last_peered":"2023-12-06T09:22:06.430186+0000","last_clean":"2023-12-06T09:22:06.430186+0000","last_became_active":"2023-12-06T09:22:04.151457+0000","last_became_peered":"2023-12-06T09:22:04.151457+0000","last_unstale":"2023-12-06T09:22:06.430186+0000","last_undegraded":"2023-12-06T09:22:06.430186+0000","last_fullsized":"2023-12-06T09:22:06.430186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:22:19.703097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.782014+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:06.782014+0000","last_peered":"2023-12-06T09:22:06.782014+0000","last_clean":"2023-12-06T09:22:06.782014+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:06.782014+0000","last_undegraded":"2023-12-06T09:22:06.782014+0000","last_fullsized":"2023-12-06T09:22:06.782014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.423236+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:06.423236+0000","last_peered":"2023-12-06T09:22:06.423236+0000","last_clean":"2023-12-06T09:22:06.423236+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:06.423236+0000","last_undegraded":"2023-12-06T09:22:06.423236+0000","last_fullsized":"2023-12-06T09:22:06.423236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:06.432197+0000","last_change":"2023-12-06T09:22:04.051938+0000","last_active":"2023-12-06T09:22:06.432197+0000","last_peered":"2023-12-06T09:22:06.432197+0000","last_clean":"2023-12-06T09:22:06.432197+0000","last_became_active":"2023-12-06T09:22:04.046554+0000","last_became_peered":"2023-12-06T09:22:04.046554+0000","last_unstale":"2023-12-06T09:22:06.432197+0000","last_undegraded":"2023-12-06T09:22:06.432197+0000","last_fullsized":"2023-12-06T09:22:06.432197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:51:40.895689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":119,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T09:22:08.745779+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:08.745779+0000","last_peered":"2023-12-06T09:22:08.745779+0000","last_clean":"2023-12-06T09:22:08.745779+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:08.745779+0000","last_undegraded":"2023-12-06T09:22:08.745779+0000","last_fullsized":"2023-12-06T09:22:08.745779+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5010000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.978999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5950000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.436999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.848000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.736999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6520000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.122}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6199999999999992}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.493}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.855}]}]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.617000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.959}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.251000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8689999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.407}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.795999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":46.590000000000003}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1239999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.824999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8260000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.848000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.701000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.616}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5269999999999992}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.926}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.253}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0410000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.558999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":463280,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4640000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.085}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.34}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.807}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.988}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.425999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":463280,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8160000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.195}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.669}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.675000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.757}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8250000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.682}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.181000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.365}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.706}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:22:22.269 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [6, 2, 5, 1] 2023-12-06T09:22:22.269 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.6', '6', '2', '5', '1'] 2023-12-06T09:22:22.269 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.6 6 2 5 1 2023-12-06T09:22:22.353 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:22.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:22.392 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:22.392 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:22.392 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:22.392 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:22.392 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:22.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:22.454 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:22.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:22.503 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:22.550 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 7.5779162790000001, 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:22.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:22.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:22:22.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:22:22.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:22.555 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:22.556 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:22.571+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:22.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:22.593 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:22.594 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:22.680 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:22.681 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:22.698 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:22.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:22.714 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:22.714 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:22.714 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:22.715 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:22.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:22.770 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:22.785+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:22.784 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:22.784 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:22.860 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:22.873+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:22.885 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:22.886 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:22.888 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:22.888 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:22.898 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:22.974 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:22.974 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:23.002 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:23.017+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:23.063 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:23.064 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:23.077+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:23.067 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:23.067 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:23.077 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:23.149 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:23.202 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:23.215+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:23.245 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:23.245 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:23.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:23.299 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:23.314+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:23.366 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:23.382+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:23.382 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:23.382 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:23.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:23.474 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:23.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:23.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-06T09:22:23.565 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:23.565 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:23.571 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:23.585+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:23.573 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:23.672 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:23.687+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:23.700 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:23.700 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:23.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:23.794 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:23.809+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:23.874 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:23.888+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:23.877 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:23.877 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:23.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:23.975 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:23.989+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:24.057 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:24.057 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:24.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:24.075 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:24.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:24.176 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:24.191+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:24.195 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:24.196 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:24.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:24.290 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:24.305+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:24.374 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:24.374 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:24.380 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:24.394+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:24.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:24.480 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:24.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:24.567 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:24.567 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:24.576 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:24.581 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:24.596+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:24.682 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:24.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:24.714 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:24.715 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:24.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:24.783 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:24.798+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:24.885 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:24.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:24.892 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:24.892 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:24.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:24.985 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:25.000+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:25.070 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:25.070 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:25.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:25.123 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:25.137+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:25.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:22:25.215 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:25.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:25.216 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:25.216 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:25.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:25.310 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:25.324+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:25.389 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:25.404+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:25.396 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:25.396 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:25.405 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:25.490 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:25.504+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:22:25.570 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T09:22:25.571 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T09:22:25.571 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:22:25.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:25.572 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:25.573 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:25.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:25.590 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:25.604+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:25.691 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:25.705+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:25.751 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:25.752 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:25.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:25.792 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:25.806+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:25.893 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:25.907+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:25.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:25.994 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:26.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:26.072 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:26.072 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:26.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:26.095 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:26.109+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:26.196 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:26.210+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:26.211 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":78,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:26.214 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:26.214 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:26.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:26.299 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:26.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:26.392 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:26.393 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:26.401 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:26.415+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:26.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:26.501 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:26.516+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:26.569 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:26.569 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:26.579 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:26.620 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:26.634+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:26.703 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:26.718+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:26.707 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:26.708 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:26.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:26.804 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:26.818+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:26.846 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:26.846 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:26.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:26.906 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:26.920+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:27.007 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:27.021+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:27.029 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:27.030 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:27.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:27.108 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:27.123+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:27.208 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:27.208 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:27.209 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:27.223+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:27.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:27.310 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:27.324+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:27.386 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:27.386 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:27.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:27.439 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:27.453+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:27.511 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:27.525+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:27.565 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:27.565 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:27.575 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:27.612 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:27.626+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:27.713 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:27.728+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:27.744 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:27.745 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:27.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:27.841 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:27.854+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:27.915 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:27.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:27.924 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:27.924 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:27.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:28.020 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:28.034+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:28.103 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:28.103 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:28.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:28.117 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:28.131+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:28.219 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:28.233+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:28.286 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:28.286 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:28.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:28.320 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:28.334+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:28.421 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:28.435+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:28.469 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:28.470 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:28.478 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:28.522 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:28.536+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:28.623 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:28.637+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:28.651 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:28.651 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:28.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:28.724 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:28.738+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:28.801 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:28.801 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:28.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:28.825 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:28.839+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:28.926 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:28.940+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:28.982 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:28.982 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:28.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:29.036 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:29.050+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:29.122 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:29.122 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:29.128 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:29.142+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:29.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:29.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:29.228 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:29.243+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:29.323 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:29.323 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:29.330 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:29.344+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:29.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:29.449 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:29.463+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:29.462 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:29.462 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:29.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:29.532 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:29.546+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:29.633 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:29.647+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:29.642 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:29.642 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:29.653 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:29.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-06T09:22:29.738 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:29.752+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:29.821 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:29.821 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:29.830 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:29.871 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:29.885+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:29.936 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:29.950+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:30.000 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.000 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:30.060 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:30.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:30.138 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:30.152+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:30.146 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.146 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:30.239 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:30.253+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:30.327 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.327 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:30.339 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:30.354+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:30.441 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:30.455+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:30.465 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.466 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.475 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:30.542 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:30.556+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:30.643 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.643 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.643 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:30.658+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:30.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:30.745 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:30.759+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:30.794 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.795 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:30.848 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:30.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:30.931 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:30.932 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:30.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:30.982 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:30.996+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:31.047 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:31.061+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:31.109 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:31.109 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:31.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:31.149 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:31.163+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:31.250 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:31.264+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:31.286 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:31.286 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:31.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:31.351 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:31.365+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:31.422 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:31.422 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:31.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:31.453 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:31.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:31.553 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:31.567+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:31.559 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:31.559 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:31.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:31.654 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:31.668+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:31.736 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:31.736 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:31.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:31.790 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:31.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:31.857 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:31.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:31.877 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:31.877 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:31.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:31.957 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:31.971+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:32.054 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:32.054 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:32.057 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:32.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:32.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:32.159 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:32.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:32.229 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:32.229 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:32.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:32.282 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:32.294+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:32.361 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:32.376+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:32.409 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:32.409 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:32.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:32.463 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:32.477+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:32.546 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:32.546 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:32.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:32.601 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:32.615+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:32.664 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:32.678+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:32.697 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:32.697 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:32.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:32.765 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:32.779+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:32.866 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:32.880+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:32.877 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:32.877 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:32.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:32.971 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:32.985+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:33.056 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:33.056 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:33.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:33.107 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:33.121+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:33.169 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:33.183+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:33.232 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:33.232 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:33.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:33.272 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:33.285+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:33.373 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:33.387+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:33.407 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:33.407 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:33.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:33.474 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:33.488+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:33.543 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:33.543 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:33.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:33.575 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:33.589+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:33.676 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:33.690+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:33.682 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:33.682 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:33.692 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:33.780 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:33.794+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:33.868 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:33.868 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:33.878 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:33.879 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:33.892+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:33.981 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:33.993+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:34.046 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:34.046 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:34.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:34.103 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:34.116+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:34.180 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:34.195+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:34.187 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:34.187 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:34.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:34.282 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:34.296+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:34.366 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:34.366 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:34.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:34.419 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:34.433+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:34.490 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:34.501+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:34.507 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:34.508 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:34.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:34.591 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:34.605+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:34.684 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:34.695 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:34.707+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:34.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:34.794 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:34.808+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:34.866 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:34.867 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:34.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:34.919 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:34.933+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:34.995 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:35.009+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:35.015 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:35.015 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:35.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:35.111 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:35.125+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:35.197 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:35.211+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:35.215 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:35.215 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:35.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:35.308 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:35.322+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:35.393 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:35.393 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:35.400 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:35.414+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:35.402 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:35.501 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:35.515+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:35.557 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:35.558 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:35.602 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:35.616+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:35.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:35.646 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:35.647 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:35.703 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:35.717+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:35.733 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:35.734 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:35.779 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:35.779 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:35.788 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:35.819 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:35.820 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:35.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:35.906 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:35.920+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:35.923 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:35.923 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:35.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:35.936 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:35.936 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:35.959 INFO:teuthology.orchestra.run.smithi088.stderr:set 2.6 pg_upmap_items mapping to [6->2,5->1] 2023-12-06T09:22:36.021 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:36.035+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:36.036 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:22:36.041 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:36.042 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:36.108 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:36.122+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:36.155 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:36.155 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:36.165 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:36.165 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:36.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:36.165 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:36.165 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:36.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:36.207 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:36.215 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:36.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:36.222 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:22:36.222 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T09:22:36.223 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T09:22:36.263 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:36.263 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:36.265 INFO:tasks.ceph.mon.a:Started 2023-12-06T09:22:36.266 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:22:36.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:36.316 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:36.330+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:36.356 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:36.357 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:36.357 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:36.357 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:36.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:36.413 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:36.427+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:36.501 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:36.501 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:36.501 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:36.501 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:36.501 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:36.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:36.511 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:36.525+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:36.612 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:36.626+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:36.694 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 21.093222286, 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:22:36.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:22:36.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:22:36.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:36.698 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:36.699 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:36.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:36.748 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:36.762+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:36.817 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:36.831+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:36.844 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:36.844 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:36.844 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:36.844 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:36.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:36.913 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:36.918 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:36.932+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:37.019 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:37.033+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:37.031 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:37.032 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:37.032 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:37.032 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:37.032 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:37.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:37.120 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:37.134+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:37.216 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:37.216 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:37.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:37.216 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:37.216 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:37.222 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:37.237+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:37.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:37.324 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:37.338+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:37.398 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:37.398 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:37.398 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:37.398 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:37.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:37.450 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:37.464+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:37.526 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:37.540+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:37.534 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:37.534 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:37.534 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:37.534 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:37.534 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:37.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:37.630 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:37.644+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:37.729 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:37.742+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:37.774 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 40.865440773000003, 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:22:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:22:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:22:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:22:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:22:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.447730655999997, 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.443546210999997, 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:22:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:22:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:22:37.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:22:37.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.358015125000001, 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:22:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:37.813 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.04403851, 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:22:37.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:22:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:22:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.042520666000001, 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:22:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:22:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.041855290999997, 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:22:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:22:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:22:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:22:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:22:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.037595119000002, 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:22:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:22:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:22:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.036961910000002, 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:22:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 38.036334807999999, 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:22:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:22:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 37.948075029999998, 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:22:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:22:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 37.947154050000002, 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:22:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:22:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 37.194776118999997, 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:22:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:22:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:22:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:22:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.324741148000001, 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:22:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:22:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.323567637, 2023-12-06T09:22:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:22:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:22:37.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.315415666, 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.305133730000001, 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.304475849999999, 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:22:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:22:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:22:37.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.303801070999999, 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.903 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 29.271443479999999, 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:22:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:37.914 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:37.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:37.916 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:37.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:38.017 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:38.031+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:38.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:22:38.088 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:38.088 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:38.089 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:38.089 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:38.098 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:38.140 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:38.154+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:38.219 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:38.233+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:38.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:22:38.298 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:38.298 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:38.298 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:38.298 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:38.298 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:38.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:38.353 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:38.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:38.420 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:38.434+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:38.501 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:38.501 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:38.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:38.501 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:38.501 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:38.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:38.531 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:38.532 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:38.546+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:38.620 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:38.621 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:38.622 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:38.636+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:38.664 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:38.664 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:38.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:38.723 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:38.737+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:38.733 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:38.825 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:38.839+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:38.840 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:38.841 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:38.841 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:38.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:38.924 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:38.925 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:38.925 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:38.939+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:39.011 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:39.027 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:39.041+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:39.039 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:39.039 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:39.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:39.103 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:39.104 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:39.128 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:39.142+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:39.180 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:22:39.200 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:39.200 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:39.203 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:39.203 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:39.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:39.254 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:39.268+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:39.331 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:39.346+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:39.339 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:39.339 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:39.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:39.434 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:39.448+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:39.520 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:39.520 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:39.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:39.535 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:39.549+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:39.625 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:54.660 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:39.627 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:54.661 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:39.636 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:39.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:39.677 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:54.707 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:39.677 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:39.677 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:39.678 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:54.750 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:39.678 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:54.752 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:39.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:39.773 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:39.785+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:39.780 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:54.853 86648== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:39.838 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:39.852+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:39.862 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:39.862 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:39.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:39.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:39.978 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:39.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:40.040 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:40.054+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:40.045 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:40.046 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:40.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:40.141 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:40.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:40.181 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:40.181 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:40.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:40.242 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:40.256+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:40.344 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:40.358+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:40.358 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:40.358 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:40.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:40.455 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:40.469+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:40.542 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:40.543 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:40.544 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:40.558+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:40.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:40.578 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:40.645 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:40.659+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:40.724 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:40.725 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:40.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:40.773 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:40.787+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:40.848 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:40.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:40.901 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:40.901 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:40.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:40.968 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:40.978+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:40.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:22:40.973 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:22:40.973 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T09:22:40.974 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T09:22:41.039 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:41.039 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:41.051 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:41.065+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:41.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:41.152 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:41.166+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:41.183 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:41.183 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:41.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:41.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:22:41.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:22:41.280 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:41.294+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:41.354 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:41.368+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:41.356 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:41.356 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:41.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:41.455 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:41.469+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:41.556 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:41.570+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:41.583 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:41.583 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:41.583 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 24, 2023-12-06T09:22:41.583 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:22:41.583 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:21:08.071156+0000", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:22:39.059866+0000", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "last_up_change": "2023-12-06T09:21:55.261571+0000", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "last_in_change": "2023-12-06T09:21:13.597313+0000", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_num": 5804032, 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_set": [ 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "nodeep-scrub", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "pglog_hardlimit", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "purged_snapdirs", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_deletes", 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "sortbitwise" 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_version": 6, 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T09:22:41.584 INFO:teuthology.orchestra.run.smithi088.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_snapshot": "", 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_max": 2, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "max_osd": 8, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "require_min_compat_client": "luminous", 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "min_compat_client": "jewel", 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "require_osd_release": "reef", 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "allow_crimson": false, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pools": [ 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 1, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": ".mgr", 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:21:54.386974+0000", 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:22:41.585 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 1, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 1, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 1, 2023-12-06T09:22:41.586 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 1, 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 1, 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:22:41.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "18", 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:22:41.588 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:22:41.589 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "options": { 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_max": 32, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_min": 1 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": {} 2023-12-06T09:22:41.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 8, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 8, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.25, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 2, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 2, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 1, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 1, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 1 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 2, 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": "rbd", 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:22:01.832571+0000", 2023-12-06T09:22:41.591 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 8, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 8, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 8, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 8, 2023-12-06T09:22:41.592 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 8, 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "21", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:22:41.593 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:22:41.594 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:22:41.595 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "options": {}, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd": {} 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 4.9800000190734863, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 4.9800000190734863, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 4.380000114440918, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 4.380000114440918, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 1, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 1, 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 1 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.596 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "db81b709-a318-4b43-beb2-bf3687f365e1", 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6805", 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6806", 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6808", 2023-12-06T09:22:41.599 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6807", 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6805/2388339328", 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6806/2388339328", 2023-12-06T09:22:41.600 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6808/2388339328", 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6807/2388339328", 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd7b0482-e237-424b-8eff-8db905713049", 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.601 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 15, 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6801", 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.602 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6802", 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6804", 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.603 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6803", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6801/3646708602", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6802/3646708602", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6804/3646708602", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6803/3646708602", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e5a571be-6b8e-4c6f-bd3e-2af30c112285", 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.605 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6810", 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6813", 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6816", 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6814", 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:22:41.607 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6810/3714168299", 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6813/3714168299", 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6816/3714168299", 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6814/3714168299", 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "d7abe19a-cc3d-411f-8533-8592e1379735", 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.608 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6809", 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.609 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6811", 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6815", 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6812", 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6809/1828251745", 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6811/1828251745", 2023-12-06T09:22:41.611 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1828251745", 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6812/1828251745", 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b9517160-d081-44f9-aea4-6d2529e02114", 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.612 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 13, 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6800", 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.614 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.614 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6801", 2023-12-06T09:22:41.614 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:41.614 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6803", 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6802", 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6800/3720978101", 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6801/3720978101", 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6803/3720978101", 2023-12-06T09:22:41.617 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6802/3720978101", 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e", 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6804", 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.619 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6805", 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6807", 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:41.620 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.621 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.621 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.621 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.625 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.625 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6806", 2023-12-06T09:22:41.625 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6804/1203998358", 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6805/1203998358", 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1203998358", 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6806/1203998358", 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e7afeca8-1363-4e9b-96f6-7fab6113d9e0", 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.627 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6808", 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.628 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6809", 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6811", 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.629 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6810", 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6808/4151455937", 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6809/4151455937", 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6811/4151455937", 2023-12-06T09:22:41.630 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6810/4151455937", 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd4b3971-3adb-44fd-b42b-b6660305beb3", 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:22:41.631 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6812", 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6813", 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6815", 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6814", 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6812/4165054852", 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6813/4165054852", 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6815/4165054852", 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6814/4165054852", 2023-12-06T09:22:41.634 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_xinfo": [ 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.635 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.229602+0000", 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.158659+0000", 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.636 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.969728+0000", 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.234010+0000", 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:50.601730+0000", 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.638 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.304649+0000", 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.639 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.671399+0000", 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.055036+0000", 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:22:41.640 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap": [], 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_items": [], 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_primaries": [], 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_temp": [], 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_temp": [], 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklist": {}, 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "range_blocklist": {}, 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profiles": { 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "default": { 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "crush-failure-domain": "osd", 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "k": "2", 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "m": "1", 2023-12-06T09:22:41.641 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin": "jerasure", 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "technique": "reed_sol_van" 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps_queue": [], 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "new_removed_snaps": [], 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "new_purged_snaps": [], 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_node_flags": {}, 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "device_class_flags": {}, 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": { 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_enabled": false, 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_bucket_count": 0, 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "degraded_stretch_mode": 0, 2023-12-06T09:22:41.642 INFO:teuthology.orchestra.run.smithi088.stdout: "recovering_stretch_mode": 0, 2023-12-06T09:22:41.643 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_bucket": 0 2023-12-06T09:22:41.643 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:41.643 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:41.645 INFO:tasks.thrashosds.thrasher:j is {'epoch': 24, 'fsid': '0c48cca3-dcc0-44bd-8eed-37fea2138550', 'created': '2023-12-06T09:21:08.071156+0000', 'modified': '2023-12-06T09:22:39.059866+0000', 'last_up_change': '2023-12-06T09:21:55.261571+0000', 'last_in_change': '2023-12-06T09:21:13.597313+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T09:21:54.386974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T09:22:01.832571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'db81b709-a318-4b43-beb2-bf3687f365e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6805', 'nonce': 2388339328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6806', 'nonce': 2388339328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6808', 'nonce': 2388339328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6807', 'nonce': 2388339328}]}, 'public_addr': '172.21.15.88:6805/2388339328', 'cluster_addr': '172.21.15.88:6806/2388339328', 'heartbeat_back_addr': '172.21.15.88:6808/2388339328', 'heartbeat_front_addr': '172.21.15.88:6807/2388339328', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'fd7b0482-e237-424b-8eff-8db905713049', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6801', 'nonce': 3646708602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6802', 'nonce': 3646708602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6804', 'nonce': 3646708602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6803', 'nonce': 3646708602}]}, 'public_addr': '172.21.15.88:6801/3646708602', 'cluster_addr': '172.21.15.88:6802/3646708602', 'heartbeat_back_addr': '172.21.15.88:6804/3646708602', 'heartbeat_front_addr': '172.21.15.88:6803/3646708602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e5a571be-6b8e-4c6f-bd3e-2af30c112285', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6810', 'nonce': 3714168299}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6813', 'nonce': 3714168299}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6816', 'nonce': 3714168299}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6814', 'nonce': 3714168299}]}, 'public_addr': '172.21.15.88:6810/3714168299', 'cluster_addr': '172.21.15.88:6813/3714168299', 'heartbeat_back_addr': '172.21.15.88:6816/3714168299', 'heartbeat_front_addr': '172.21.15.88:6814/3714168299', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd7abe19a-cc3d-411f-8533-8592e1379735', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6809', 'nonce': 1828251745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6811', 'nonce': 1828251745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6815', 'nonce': 1828251745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6812', 'nonce': 1828251745}]}, 'public_addr': '172.21.15.88:6809/1828251745', 'cluster_addr': '172.21.15.88:6811/1828251745', 'heartbeat_back_addr': '172.21.15.88:6815/1828251745', 'heartbeat_front_addr': '172.21.15.88:6812/1828251745', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b9517160-d081-44f9-aea4-6d2529e02114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 3720978101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6801', 'nonce': 3720978101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6803', 'nonce': 3720978101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 3720978101}]}, 'public_addr': '172.21.15.132:6800/3720978101', 'cluster_addr': '172.21.15.132:6801/3720978101', 'heartbeat_back_addr': '172.21.15.132:6803/3720978101', 'heartbeat_front_addr': '172.21.15.132:6802/3720978101', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1203998358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6805', 'nonce': 1203998358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6807', 'nonce': 1203998358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1203998358}]}, 'public_addr': '172.21.15.132:6804/1203998358', 'cluster_addr': '172.21.15.132:6805/1203998358', 'heartbeat_back_addr': '172.21.15.132:6807/1203998358', 'heartbeat_front_addr': '172.21.15.132:6806/1203998358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7afeca8-1363-4e9b-96f6-7fab6113d9e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 4151455937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6809', 'nonce': 4151455937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6811', 'nonce': 4151455937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 4151455937}]}, 'public_addr': '172.21.15.132:6808/4151455937', 'cluster_addr': '172.21.15.132:6809/4151455937', 'heartbeat_back_addr': '172.21.15.132:6811/4151455937', 'heartbeat_front_addr': '172.21.15.132:6810/4151455937', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fd4b3971-3adb-44fd-b42b-b6660305beb3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 4165054852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6813', 'nonce': 4165054852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6815', 'nonce': 4165054852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 4165054852}]}, 'public_addr': '172.21.15.132:6812/4165054852', 'cluster_addr': '172.21.15.132:6813/4165054852', 'heartbeat_back_addr': '172.21.15.132:6815/4165054852', 'heartbeat_front_addr': '172.21.15.132:6814/4165054852', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.229602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.158659+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.969728+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.234010+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:50.601730+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.304649+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.671399+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.055036+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T09:22:41.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:22:41.646 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:41.646 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:41.647 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:41.647 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:41.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:41.647 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:41.647 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:41.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:41.657 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:41.671+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:41.726 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:41.758 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:41.772+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:41.834 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:41.858 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:41.872+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:41.872 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:41.873 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:41.873 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:41.873 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:41.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:41.926 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:41.927 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:41.960 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:41.974+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:42.012 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:42.013 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:42.014 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:42.014 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:42.014 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:42.014 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:42.014 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:42.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:42.061 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:42.075+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:42.135 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:42.135 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:42.162 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:42.176+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:42.221 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:42.221 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:42.253 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 45.344571141999999, 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:22:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.926861025000001, 2023-12-06T09:22:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:22:42.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:22:42.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:22:42.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.922676580000001, 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.267 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.837145493999998, 2023-12-06T09:22:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.523168878999996, 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:22:42.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:22:42.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:22:42.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:22:42.282 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.521651034999998, 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.283 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.520985660000001, 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.288 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:22:42.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:22:42.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:22:42.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.516725487999999, 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.296 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:22:42.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:22:42.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:22:42.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:42.301 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.516092278999999, 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:22:42.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:22:42.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.515465177000003, 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.307 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:22:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:22:42.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:22:42.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.427205399000002, 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.312 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.426284418999998, 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:22:42.317 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:42.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:22:42.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:22:42.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.322 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 41.673906488, 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:22:42.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:22:42.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.803871516999997, 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:22:42.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:22:42.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.802698006, 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.333 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:22:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:22:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:22:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:22:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.794546035000003, 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:22:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.784264098999998, 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:22:42.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.783606218999999, 2023-12-06T09:22:42.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:22:42.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:22:42.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.354 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.782931439999999, 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 33.750573848999998, 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:42.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:22:42.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:22:42.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:42.365 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:42.365 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:42.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:42.367 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:42.368 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:42.368 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:42.380+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:42.419 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:22:42.419 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:22:42.431 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2023-12-06T09:22:41.928122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217800,"kb_used_data":1640,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756920,"statfs":{"total":773094113280,"available":772871086080,"internally_reserved":0,"allocated":1679360,"data_stored":1370816,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001348"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-06T09:22:36.151438+0000","last_change":"2023-12-06T09:22:04.084324+0000","last_active":"2023-12-06T09:22:36.151438+0000","last_peered":"2023-12-06T09:22:36.151438+0000","last_clean":"2023-12-06T09:22:36.151438+0000","last_became_active":"2023-12-06T09:22:04.075169+0000","last_became_peered":"2023-12-06T09:22:04.075169+0000","last_unstale":"2023-12-06T09:22:36.151438+0000","last_undegraded":"2023-12-06T09:22:36.151438+0000","last_fullsized":"2023-12-06T09:22:36.151438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:06.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T09:22:17.855489+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:17.855489+0000","last_peered":"2023-12-06T09:22:17.855489+0000","last_clean":"2023-12-06T09:22:17.855489+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:17.855489+0000","last_undegraded":"2023-12-06T09:22:17.855489+0000","last_fullsized":"2023-12-06T09:22:17.855489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-06T09:22:36.162373+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:36.162373+0000","last_peered":"2023-12-06T09:22:36.162373+0000","last_clean":"2023-12-06T09:22:36.162373+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:36.162373+0000","last_undegraded":"2023-12-06T09:22:36.162373+0000","last_fullsized":"2023-12-06T09:22:36.162373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T09:22:17.804251+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:17.804251+0000","last_peered":"2023-12-06T09:22:17.804251+0000","last_clean":"2023-12-06T09:22:17.804251+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:17.804251+0000","last_undegraded":"2023-12-06T09:22:17.804251+0000","last_fullsized":"2023-12-06T09:22:17.804251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-06T09:22:36.155471+0000","last_change":"2023-12-06T09:22:04.156624+0000","last_active":"2023-12-06T09:22:36.155471+0000","last_peered":"2023-12-06T09:22:36.155471+0000","last_clean":"2023-12-06T09:22:36.155471+0000","last_became_active":"2023-12-06T09:22:04.151457+0000","last_became_peered":"2023-12-06T09:22:04.151457+0000","last_unstale":"2023-12-06T09:22:36.155471+0000","last_undegraded":"2023-12-06T09:22:36.155471+0000","last_fullsized":"2023-12-06T09:22:36.155471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:22:19.703097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-06T09:22:36.295086+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:36.295086+0000","last_peered":"2023-12-06T09:22:36.295086+0000","last_clean":"2023-12-06T09:22:36.295086+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:36.295086+0000","last_undegraded":"2023-12-06T09:22:36.295086+0000","last_fullsized":"2023-12-06T09:22:36.295086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-06T09:22:36.153765+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:36.153765+0000","last_peered":"2023-12-06T09:22:36.153765+0000","last_clean":"2023-12-06T09:22:36.153765+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:36.153765+0000","last_undegraded":"2023-12-06T09:22:36.153765+0000","last_fullsized":"2023-12-06T09:22:36.153765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-06T09:22:36.164384+0000","last_change":"2023-12-06T09:22:04.051938+0000","last_active":"2023-12-06T09:22:36.164384+0000","last_peered":"2023-12-06T09:22:36.164384+0000","last_clean":"2023-12-06T09:22:36.164384+0000","last_became_active":"2023-12-06T09:22:04.046554+0000","last_became_peered":"2023-12-06T09:22:04.046554+0000","last_unstale":"2023-12-06T09:22:36.164384+0000","last_undegraded":"2023-12-06T09:22:36.164384+0000","last_fullsized":"2023-12-06T09:22:36.164384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:51:40.895689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":120,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T09:22:17.903257+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:17.903257+0000","last_peered":"2023-12-06T09:22:17.903257+0000","last_clean":"2023-12-06T09:22:17.903257+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:17.903257+0000","last_undegraded":"2023-12-06T09:22:17.903257+0000","last_fullsized":"2023-12-06T09:22:17.903257+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509452,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":74037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.033000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9960000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.686}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.613}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.212}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.994}]}]},{"osd":2,"up_from":15,"seq":64424509452,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":74037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5030000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.467000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.609}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.082000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.788}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.213999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.853000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":74037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0940000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.762}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.846}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.370999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.526999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.457000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.184999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":74037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.021000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.962}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.552}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.06}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.941000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.856999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.908999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":69747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.273}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.917999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.396000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.146000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.696999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.622}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.013000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":467587,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9980000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4209999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.558999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.724}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.652000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":467587,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.491}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1989999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.292999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.463000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.510999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.213999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":69747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9830000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.061}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7160000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.07}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.917000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.052}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.84}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:22:42.432 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [7, 0] 2023-12-06T09:22:42.432 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '7', '0'] 2023-12-06T09:22:42.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 7 0 2023-12-06T09:22:42.473 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:42.487+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:42.563 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:42.563 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:42.563 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:42.563 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:42.568 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:42.582+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:42.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:42.600 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:22:42.668 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:42.683+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:42.731 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:42.731 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:42.731 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:42.732 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:42.732 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:42.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:42.784 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:42.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:42.870 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:42.885+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:42.913 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:42.914 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:42.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:42.914 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:42.914 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:42.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:42.972 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:42.986+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:43.075 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:43.075 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:43.075 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:43.075 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:43.075 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:43.089+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:43.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:43.177 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:43.191+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:43.254 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:43.254 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:43.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:43.312 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:43.325+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:43.379 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:43.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:43.397 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 28.436296605999999, 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:43.398 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:43.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:22:43.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:43.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:43.403 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:43.403 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:43.407 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:43.480 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:43.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:43.538 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:43.538 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:43.538 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:43.538 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:43.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:43.589 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:43.603+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:43.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:43.672 INFO:teuthology.orchestra.run.smithi088.stderr:set 2.7 pg_upmap mapping to [7,0] 2023-12-06T09:22:43.684 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:43.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:43.735 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:43.735 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:43.735 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:43.735 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:43.735 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:43.743 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:43.787 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:43.801+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:43.892 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:43.903+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:43.896 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:43.896 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:43.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:43.896 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:43.896 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:43.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:43.984 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:43.998+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:44.084 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:44.085 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:44.085 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:44.085 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:44.085 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:44.099+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:44.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:44.189 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:44.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:44.226 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:44.227 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:44.227 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:44.227 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:44.227 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:44.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:44.283 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:44.287 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:44.301+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:44.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 45.700440516, 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:22:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:22:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:22:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:22:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:22:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 45.005986647, 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:22:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:22:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.904158066999997, 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:22:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:22:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:22:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.563094171000003, 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:22:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.554267889999998, 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:22:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:22:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:22:44.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.553646217999997, 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:22:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:22:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.545699737, 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:22:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:22:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:22:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.545225719000001, 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:22:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:22:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:22:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.544771363999999, 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:22:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 44.160779112999997, 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:22:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 43.775629707999997, 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:22:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:22:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 43.763354397000001, 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:22:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:22:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:22:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:22:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:22:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 43.761868819, 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:22:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:22:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:22:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:22:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.855331661000001, 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:22:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.850199357999998, 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.849373327999999, 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.502 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.503 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.504 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:22:44.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:22:44.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.505 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.505 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.505 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:22:44.505 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.506 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.846803866999998, 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.507 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.508 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.509 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.510 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.846058397, 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.511 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.512 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.513 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:22:44.514 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.515 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.845252326000001, 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:44.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:22:44.517 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:22:44.518 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:22:44.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 35.834147874000003, 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:22:44.520 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:22:44.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:22:44.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:22:44.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:44.524 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:44.524 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:44.430+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:44.525 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:44.567 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:44.581+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:44.614 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:22:44.615 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.286 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:44.615 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:44.615 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:44.659 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:44.673+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:44.700 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:44.700 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:44.700 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:44.700 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:44.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:44.725 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:44.757 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:44.771+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:44.827 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:44.828 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:44.832 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:44.844+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:44.847 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:44.847 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:44.847 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:44.847 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:44.847 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:44.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:44.883 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.555 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:44.918 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:44.918 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:44.950 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:44.962+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:45.026 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:45.026 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:45.028 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:45.028 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:45.033 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:45.047+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:45.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:45.122 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:45.134 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:45.148+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:45.221 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:45.222 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:45.222 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:45.222 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:45.226 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:45.230 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:45.275 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:45.288+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:45.333 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:45.334 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:45.337 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:45.351+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:45.371 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:45.371 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:45.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:45.471 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:45.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:45.539 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:45.553+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:45.554 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:45.555 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:45.564 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:45.639 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:45.653+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:45.698 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:45.698 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:45.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:45.749 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:45.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:45.840 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:45.855+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:45.888 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:45.943 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:22:45.943 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:45.957+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:46.020 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:46.021 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:46.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:46.044 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:46.058+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:46.148 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:46.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:46.168 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:46.168 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:46.183 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:46.247 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:46.261+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:46.348 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:46.362+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:46.354 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:46.354 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:46.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:46.449 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:46.463+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:46.539 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:46.539 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:46.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:46.551 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:46.565+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:46.650 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:46.664+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:46.682 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:46.683 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:46.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:46.753 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:46.767+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:46.854 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:46.868+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:46.861 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:46.861 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:46.870 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:46.955 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:46.970+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:47.002 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:47.002 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:47.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:47.057 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:47.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:47.146 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:47.146 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:47.156 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:47.157 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:47.171+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:47.260 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:47.274+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:47.293 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:47.293 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:47.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:47.303 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:47.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:22:47.362 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:47.376+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:47.463 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:47.478+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:47.487 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:47.488 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:47.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:47.565 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:47.579+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:47.611 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:47.627 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:47.627 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:47.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:47.665 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:47.680+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:47.724 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:47.724 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:47.766 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:47.780+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:47.805 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:47.805 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:47.813 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:47.814 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:47.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:47.867 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:47.881+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:47.901 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:47.902 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:47.959 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:22:47.970 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:47.971 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:47.985+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:48.003 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:48.003 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:48.004 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:48.004 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:48.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:48.004 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:48.004 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:48.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:48.107 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:48.107 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:48.108 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:48.121+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:48.172 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:48.186+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:48.200 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:48.200 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:48.201 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:48.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:48.210 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:48.210 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:48.273 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:48.287+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:48.298 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:48.299 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:48.374 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:48.388+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:48.386 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:48.386 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:48.386 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:48.386 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:48.386 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:48.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:48.475 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:48.489+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:48.565 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 49.857154512000001, 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:48.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:22:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:22:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:22:48.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:22:48.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 49.162700643000001, 2023-12-06T09:22:48.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:22:48.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:22:48.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:22:48.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:22:48.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:22:48.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:22:48.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:22:48.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 49.060872062999998, 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:22:48.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:22:48.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:22:48.587 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.719808166999997, 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:22:48.589 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:22:48.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:22:48.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.591 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.591 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.710981885999999, 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.596 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.710360213999998, 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:22:48.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:22:48.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:22:48.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.702413733, 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:22:48.601 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.602 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.602 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:22:48.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:22:48.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.701939715000002, 2023-12-06T09:22:48.606 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.610 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.70148536, 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.612 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:22:48.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:22:48.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:22:48.617 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.618 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.618 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.317493108999997, 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.622 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:22:48.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:22:48.623 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 47.932343703999997, 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 47.920068393000001, 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:22:48.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:22:48.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.634 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 47.918582815000001, 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:48.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:48.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:22:48.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:22:48.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:22:48.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.012045657000002, 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:22:48.642 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:22:48.646 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.006913353999998, 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.006087323999999, 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:22:48.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.003517862999999, 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:22:48.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:22:48.661 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.002772393000001, 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:48.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:22:48.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 40.001966322000001, 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:48.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:22:48.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 39.990861870000003, 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:48.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:48.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:22:48.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:48.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:22:48.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:22:48.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:48.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:48.680 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:48.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:48.682 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:48.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:48.693 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:22:48.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:22:48.694 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-12-06T09:22:48.695 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.158708 2023-12-06T09:22:48.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.15870822467014578 2023-12-06T09:22:48.782 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:48.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:48.872 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:48.872 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:48.872 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:48.883 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:48.897+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:48.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:48.984 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:48.998+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:49.056 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:49.056 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:49.056 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:49.056 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:49.057 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:49.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:49.109 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:49.123+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:49.186 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:49.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:49.198 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:49.199 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:49.199 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:49.199 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:49.199 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:49.208 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:49.279 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:22:49.287 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:49.301+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:49.384 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:49.385 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:49.390 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:49.401+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:49.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:49.489 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:49.503+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:49.566 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:49.566 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:49.566 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:49.566 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:49.566 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:49.578 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:49.619 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:49.633+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:49.686 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:13.358 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:49.691 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:49.704+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:49.739 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:13.412 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 34.159539741000003, 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:22:49.751 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:22:49.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:22:49.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:49.755 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:49.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:49.802 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:49.816+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:49.894 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:49.908+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:49.900 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:49.901 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:49.901 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:49.901 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:49.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:49.999 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:50.009+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:50.010 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:13.682 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:22:50.039 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:50.039 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:50.039 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:50.039 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:50.039 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:50.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:50.095 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:50.109+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:50.183 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:50.183 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:50.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:50.183 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:50.183 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:50.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:50.197 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:50.210+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:50.299 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:50.311+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:50.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:22:50.389 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:50.390 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:50.390 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:50.390 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:50.398 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:50.412+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:50.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:50.498 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:50.513+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:50.541 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:50.541 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:50.542 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:50.542 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:50.542 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:50.551 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:50.600 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:50.614+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:50.701 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:50.715+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 53.840981970999998, 2023-12-06T09:22:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:22:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:22:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:22:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.423271853999999, 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.761 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:22:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:22:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.419087408999999, 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:22:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:22:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:22:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.333556323000003, 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:22:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:22:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.019579708000002, 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:22:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:22:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:22:50.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.786 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.018061864000003, 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:22:50.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:22:50.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:22:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:22:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.017396488999999, 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.793 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:22:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:22:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:22:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.013136316999997, 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:22:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:22:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:22:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:22:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.012503107999997, 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:22:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:22:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:22:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:22:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:22:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.011876006000001, 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.923616228, 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:22:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:22:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.922695247999997, 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:22:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:22:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.170317316999999, 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:22:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:22:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:22:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:22:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.300282346000003, 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.299108834999998, 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:22:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.290956864000002, 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:22:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:22:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:22:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:22:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.280674928000003, 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:22:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:22:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:22:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.860 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.280017047999998, 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:50.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:22:50.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:22:50.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.866 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.279342268999997, 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:22:50.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:22:50.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:22:50.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.872 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 42.246984677999997, 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:50.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:50.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:50.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:50.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:22:50.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:22:50.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.879 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.879 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.879 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:50.879 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:50.879 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:50.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:50.881 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:50.881 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:50.881 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:50.895+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:50.937 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:50.937 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:50.981 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:50.995+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:50.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:51.033 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.3 primary-affinity to 0.158708 (8104012) 2023-12-06T09:22:51.048 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:51.048 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:51.082 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:51.096+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:51.102 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:51.102 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:51.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:51.166 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:51.166 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:51.258 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:51.271+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:51.270 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:51.271 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:51.284 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:51.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:51.288 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:51.288 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:51.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:22:51.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:51.370 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:51.404 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:51.405 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:51.419+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:51.541 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:51.542 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:51.645 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:51.646 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:51.659+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:51.679 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:51.690+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:51.708 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:51.722+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:51.745 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:51.745 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:51.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:51.759 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:51.774 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:22:51.774 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T09:22:51.774 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:22:51.815 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:51.823+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:51.891 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:22:51.890+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86626) UID: 0 2023-12-06T09:22:51.891 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:22:51.892+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T09:22:51.907 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:51.907 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:51.910 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:51.924+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:51.917 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:52.011 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:52.025+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:52.094 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:52.095 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:52.107 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:52.112 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:52.126+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:52.213 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:52.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:52.278 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:52.279 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:52.288 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:52.330 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:52.344+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:52.406 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:07.480 86648== 2023-12-06T09:22:52.407 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:07.480 86648== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:22:52.416 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:52.430+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:52.417 INFO:tasks.ceph.mon.c.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:22:52.418 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:52.418 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:52.427 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:52.516 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:52.531+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:52.556 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:52.557 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:52.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:52.618 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:52.632+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:52.694 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:52.694 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:52.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:52.748 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:52.762+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:52.820 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:52.834+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:52.834 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:52.834 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:52.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:52.922 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:52.935+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:53.012 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.012 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:53.022 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:53.036+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:53.123 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:53.137+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:53.148 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.148 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:53.226 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:53.238+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:53.288 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.288 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:53.342 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:53.356+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:53.425 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.426 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.426 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:53.440+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:53.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:53.531 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:53.545+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:53.652 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:53.666+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:53.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:22:53.733 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:53.747+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:53.785 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.785 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.795 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:53.851 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:53.865+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:53.912 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:22:53.912 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:22:53.924 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:53.924 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:53.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:53.934 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:53.948+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:53.995 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:22:53.995 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:22:54.060 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:22:54.060 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:22:54.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:54.078 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:22:54.079 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:22:54.079 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:54.093+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:54.136 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:54.150+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:54.181 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:22:54.181 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:22:54.214 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:54.215 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:54.215 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:54.215 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:54.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:54.241 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:54.251+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:54.274 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:22:54.274 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:22:54.338 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:54.352+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:54.362 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:22:54.362 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:22:54.396 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:54.397 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:54.397 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:54.397 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:54.397 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:54.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:54.446 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:54.460+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:54.452 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:22:54.453 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:22:54.543 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:22:54.544 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:22:54.566 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:54.577+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:54.576 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:54.577 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:54.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:54.577 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:54.577 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:54.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:54.646 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:54.656+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:54.743 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:54.757+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:54.762 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:54.762 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:54.762 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:54.762 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:54.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:54.848 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:54.858+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:54.901 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:54.901 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:54.901 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:54.901 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:54.901 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:54.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:54.945 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:54.959+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:54.998 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T09:22:55.046 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:55.061+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:55.134 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:55.134 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:55.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:55.134 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:55.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 58.226956430999998, 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.135 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:22:55.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:22:55.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:22:55.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:22:55.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.809246313999999, 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:22:55.143 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:22:55.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:22:55.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:22:55.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:22:55.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.153 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.805061868999999, 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:22:55.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:22:55.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:22:55.161 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.719530783000003, 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:22:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:22:55.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:22:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:22:55.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.405554168000002, 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.167 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:55.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:22:55.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:22:55.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.404036324000003, 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.173 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:22:55.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:22:55.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:22:55.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.403370948999999, 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:22:55.179 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:22:55.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.399110776999997, 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.186 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.398477567999997, 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:22:55.191 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:55.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:22:55.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.397850466000001, 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.309590688, 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:22:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:22:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.308669707999996, 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:22:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 54.556291776999998, 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:22:55.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.686256806000003, 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:22:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:22:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.685083294999998, 2023-12-06T09:22:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:22:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:22:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:22:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.676931324000002, 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.666649388000003, 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:22:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:22:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.665991507999998, 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:22:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:22:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:22:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.665316728999997, 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:22:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:22:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:22:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 46.632959137999997, 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:22:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:55.275 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:55.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:55.277 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:55.291+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:55.378 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:55.392+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:55.446 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:55.446 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:55.446 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:55.446 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:55.459 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:55.479 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:55.493+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:55.580 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:55.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:55.585 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:55.585 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:55.585 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:55.585 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:55.585 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:55.596 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:55.682 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:55.695+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:55.728 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:55.728 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:55.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:55.729 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:55.729 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:55.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:22:55.783 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:55.797+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:55.884 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:55.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:55.910 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:55.910 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:55.910 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:55.910 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:55.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:22:55.986 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:56.000+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:56.046 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:56.046 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:56.046 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:56.046 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:56.046 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:56.048 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:22:56.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:22:56.048 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2023-12-06T09:22:56.048 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T09:22:56.048 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T09:22:56.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2023-12-06T09:22:56.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:22:56.099 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:56.112+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:56.189 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:56.203+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:56.192 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 41.231199566000001, 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.193 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:22:56.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:56.196 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:56.197 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:56.197 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T09:22:56.197 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:56.202 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:22:56.290 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:56.304+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:56.335 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:56.335 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:56.336 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:56.336 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:56.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:22:56.391 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:56.405+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:56.496 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:56.509+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:56.513 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:56.513 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:56.513 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:56.513 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:56.513 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:56.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:22:56.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:22:56.607 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:56.621+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:56.692 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:56.693 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:56.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:56.693 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:56.693 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:56.698 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:56.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:56.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:22:56.799 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:56.813+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:56.869 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:56.869 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:56.879 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:22:56.900 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:56.914+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:57.001 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:57.015+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:57.010 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:57.010 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:57.010 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:57.010 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:57.010 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:57.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:22:57.103 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:57.117+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:57.204 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:57.218+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 58.521615005000001, 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.231 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:22:57.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:22:57.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:22:57.239 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.827161136000001, 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:57.243 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.725332555999998, 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.245 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.246 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:22:57.246 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.248 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:22:57.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:22:57.248 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.249 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.384268659999996, 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.253 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.375442378999999, 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:57.254 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:22:57.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:22:57.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:22:57.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:22:57.258 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.258 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.258 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.258 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.258 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.374820706999998, 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.259 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.260 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.261 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:22:57.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.366874226, 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:22:57.263 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.264 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.265 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.266 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:22:57.267 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.366400208000002, 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:22:57.269 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.270 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.272 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.365945852999999, 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:22:57.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:22:57.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 56.981953601999997, 2023-12-06T09:22:57.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.279 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:22:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:22:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.280 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.281 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:22:57.282 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 56.596804196999997, 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:22:57.284 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:22:57.285 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.286 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.287 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 56.584528886000001, 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.289 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:22:57.290 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:22:57.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.291 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 56.583043308000001, 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.295 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.296 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:57.296 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.296 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:22:57.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:57.300 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.676506150000002, 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.302 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.303 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:22:57.304 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.306 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.671373846999998, 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.307 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.308 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:22:57.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.670547816999999, 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:22:57.312 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.313 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.314 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:22:57.316 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.667978355999999, 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:22:57.317 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.318 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.318 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.318 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.318 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:22:57.318 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:22:57.318 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.319 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.667232886000001, 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:22:57.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:22:57.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.326 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.666426815000001, 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:57.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:22:57.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:22:57.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:22:57.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:22:57.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 48.655322363000003, 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:57.333 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:22:57.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:57.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:57.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:57.338 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:57.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:22:57.346 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:57.359+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:57.446 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:57.461+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:57.516 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:57.517 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:57.517 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:57.517 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:57.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:22:57.570 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:57.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:57.651 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:57.662+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:57.652 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:57.653 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:57.653 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:57.653 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:57.653 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:57.667 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:22:57.751 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:57.765+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:57.839 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:57.840 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:57.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:57.840 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:22:57.840 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:57.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:22:57.876 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:22:57.876 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T09:22:57.876 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T09:22:57.876 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:22:57.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:22:57.891 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:57.904+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:57.952 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:57.966+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:58.024 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:22:58.024 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:58.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:22:58.053 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:58.067+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:58.155 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:58.169+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:58.164 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:58.164 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:22:58.164 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:22:58.164 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:22:58.164 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:58.174 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:22:58.256 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:58.270+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:58.345 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 42.754502129000002, 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:22:58.346 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:22:58.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:22:58.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:22:58.349 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:22:58.350 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:22:58.355 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:22:58.360 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:22:58.370+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:22:58.457 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:58.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:58.490 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:58.490 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:58.491 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:58.491 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:58.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:22:58.534 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:22:58.589 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:22:58.603+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:22:58.660 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:58.674+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:58.670 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:58.670 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:58.670 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:58.670 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:58.670 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:58.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:22:58.764 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:58.778+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:58.850 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:58.850 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:58.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:58.850 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:22:58.850 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:58.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:22:58.862 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:22:58.876+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:22:58.963 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:58.977+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:59.024 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:59.024 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:59.024 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:59.024 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:59.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:22:59.064 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:59.078+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:59.169 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:22:59.179+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:22:59.169 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:59.169 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:59.169 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:59.169 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:59.169 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:59.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:22:59.266 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:59.280+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:59.369 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:59.382+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:22:59.399 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 62.494695309000001, 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:59.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:22:59.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:22:59.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:22:59.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:22:59.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:22:59.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:22:59.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:22:59.408 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.408 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.408 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.076985192000002, 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.409 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:22:59.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:22:59.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:22:59.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:22:59.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:22:59.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:22:59.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:22:59.430 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.072800747000002, 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.431 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:22:59.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:22:59.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:22:59.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:59.441 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.987269660999999, 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:22:59.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.443 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:59.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:22:59.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:22:59.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:22:59.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:22:59.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:22:59.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:22:59.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:22:59.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.673293045999998, 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:22:59.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:22:59.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:22:59.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:22:59.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.671775201999999, 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.462 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:22:59.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:22:59.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.671109827000002, 2023-12-06T09:22:59.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:22:59.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:22:59.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.473 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.666849655, 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:22:59.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:22:59.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:22:59.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:22:59.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:22:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.666216446, 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.479 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:22:59.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:22:59.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.484 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.665589343999997, 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:59.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:59.490 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.577329566000003, 2023-12-06T09:22:59.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:22:59.491 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.491 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:22:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:22:59.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:22:59.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:59.497 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.576408585999999, 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:22:59.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:22:59.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:22:59.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:22:59.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:22:59.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:22:59.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.506 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 58.824030655000001, 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:22:59.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:22:59.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.953995683999999, 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:22:59.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:22:59.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:22:59.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:22:59.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.521 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.952822173000001, 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:22:59.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:22:59.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.944670201999998, 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:22:59.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:22:59.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.533 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.934388265999999, 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:22:59.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:22:59.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:22:59.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.539 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.933730386000001, 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:22:59.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.541 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:59.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:22:59.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:22:59.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:22:59.545 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.933055607, 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.546 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:22:59.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:22:59.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:22:59.552 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 50.900698016, 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:22:59.553 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:22:59.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:22:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:22:59.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:22:59.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:22:59.559 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:22:59.559 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:59.560 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:22:59.561 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:59.574+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:22:59.665 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:22:59.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:22:59.730 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:59.730 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:59.730 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:22:59.730 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:59.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:22:59.762 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:22:59.777+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:22:59.870 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:22:59.877+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:22:59.871 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:22:59.871 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:22:59.871 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:22:59.871 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:22:59.871 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:22:59.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:22:59.964 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:22:59.978+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:00.008 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:00.008 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:00.008 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:00.008 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:00.008 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:00.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:00.065 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:00.080+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:00.172 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:00.186+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:00.186 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:00.186 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:00.186 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:00.187 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:00.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:00.267 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:00.281+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:00.365 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:00.365 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:00.366 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:00.366 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:00.366 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:00.368 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:00.382+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:00.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:00.469 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:00.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:00.504 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 45.543448587, 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:00.505 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:00.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:00.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:00.509 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:00.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:00.570 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:00.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:00.659 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:00.660 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:00.660 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:00.660 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:00.670 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:00.671 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:00.684+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:00.771 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:00.785+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:00.800 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:00.800 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:00.800 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:00.800 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:00.800 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:00.810 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:00.895 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:00.909+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:00.974 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:00.988+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:00.980 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:00.980 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:00.980 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:00.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:01.074 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:01.088+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:01.158 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:01.159 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:01.159 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:01.159 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:01.168 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:01.175 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:01.190+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:01.208 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:01.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:01.209 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-12-06T09:23:01.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T09:23:01.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T09:23:01.276 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:01.290+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:01.343 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:01.344 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:01.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:01.395 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:01.409+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:01.479 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:01.493+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:01.520 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:01.520 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:01.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 62.815133850000002, 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:01.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.524 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 62.120679981000002, 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:01.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:01.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 62.018851400999999, 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.531 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.533 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:01.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:01.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:01.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.547 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:01.547 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:01.547 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.677787504999998, 2023-12-06T09:23:01.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:01.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.551 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.668961224, 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:01.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:01.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.668339551999999, 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.558 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.560 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:01.560 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:01.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.562 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:01.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.660393071000001, 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:01.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.659919053000003, 2023-12-06T09:23:01.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:01.571 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.571 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.573 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.659464698000001, 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.578 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.578 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.580 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:01.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:01.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:01.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 61.275472446999999, 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.585 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.587 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.588 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:01.589 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:01.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 60.890323041999999, 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.591 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:01.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:01.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:01.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:01.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 60.878047731000002, 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:01.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 60.876562153000002, 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.606 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.970024995000003, 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:01.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:01.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.964892691999999, 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:01.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:01.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:01.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:01.620 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.964066662, 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.627 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.961497201, 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:01.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:01.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:01.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.960751731000002, 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:01.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.959945660000002, 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:01.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:01.645 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 52.948841207999997, 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:01.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:01.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:01.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:01.649 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:01.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:01.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:01.655 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:01.665+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:01.753 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:01.768+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:01.822 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:01.822 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:01.822 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:01.822 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:01.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:01.874 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:01.889+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:01.955 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:01.970+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:02.007 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:02.008 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:02.008 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:02.008 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:02.008 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:02.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:02.059 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:02.073+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:02.160 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:02.174+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:02.185 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:02.186 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:02.186 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:02.186 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:02.186 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:02.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:02.263 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:02.276+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:02.331 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:02.331 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:02.331 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:02.331 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:02.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:02.380 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:02.395+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:02.462 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:02.462 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:02.462 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:02.462 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:02.462 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:02.470 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:02.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:02.471 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:02.573 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:02.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 47.050955483999999, 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:02.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:02.646 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:02.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:02.693 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:02.707+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:02.777 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:02.792+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:02.781 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:02.781 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:02.781 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:02.781 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:02.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:02.877 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:02.891+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:02.922 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:02.922 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:02.922 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:02.922 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:02.922 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:02.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:02.978 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:02.993+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:03.080 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:03.095+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:03.104 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:03.104 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:03.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:03.104 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:03.104 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:03.109 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:03.181 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:03.196+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:03.236 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:03.237 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:03.237 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:03.237 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:03.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:03.297 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:03.311+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:03.380 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:03.380 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:03.380 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:03.380 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:03.380 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:03.384 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:03.397+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:03.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:03.484 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:03.499+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:03.585 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:03.600+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 66.713213353, 2023-12-06T09:23:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 64.295503236000002, 2023-12-06T09:23:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 64.291318790999995, 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 64.205787705000006, 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:03.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:03.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:03.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.891811089999997, 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:03.658 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.890293245999999, 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:03.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:03.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.889627871000002, 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.664 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:03.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.885367699, 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:03.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:03.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.88473449, 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.675 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:03.680 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.884107387999997, 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.685 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.795847610000003, 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:03.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.794926629999999, 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:03.691 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:03.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:03.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.042548699000001, 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:03.697 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:03.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.702 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.172513727999998, 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.171340217000001, 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.708 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:03.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.163188245999997, 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:03.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:03.718 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.152906309999999, 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:03.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:03.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.15224843, 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.724 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:03.729 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.151573651, 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 55.119216059999999, 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:03.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:03.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:03.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:03.739 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.740 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.740 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.740 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:03.740 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:03.740 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:03.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:03.742 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:03.756+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:03.856 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:03.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:03.870 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:03.870 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:03.870 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:03.870 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:03.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:03.964 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:03.978+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:04.045 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:04.059+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:04.048 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:04.048 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:04.048 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:04.048 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:04.048 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:04.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:04.146 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:04.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:04.226 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:04.226 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:04.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:04.226 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:04.226 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:04.239 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:04.280 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:04.294+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:04.353 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:04.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:04.417 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:04.417 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:04.417 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:04.417 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:04.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:04.454 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:04.468+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:04.555 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:04.568+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:04.596 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:04.596 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:04.596 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:04.596 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:04.596 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:04.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:04.656 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:04.670+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:04.739 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 49.774119122000002, 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:04.743 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:04.745 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:04.790 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:04.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:04.862 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:04.876+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:04.885 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:04.886 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:04.886 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:04.886 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:04.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:04.979 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:04.993+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:05.064 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:05.078+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:05.064 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:05.065 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:05.065 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:05.065 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:05.065 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:05.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:05.165 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:05.179+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:05.241 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:05.241 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:05.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:05.241 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:05.241 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:05.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:05.291 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:05.305+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:05.367 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:05.381+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:05.417 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:05.417 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:05.418 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:05.418 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:05.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:05.469 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:05.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:05.569 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:05.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:05.594 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:05.595 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:05.595 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:05.595 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:05.595 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:05.604 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:05.671 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:05.685+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:05.770 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:05.770 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 67.065583894, 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:05.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.774 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:05.774 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:05.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:05.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:05.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:05.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:05.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:05.780 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 66.371130024999999, 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:05.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:05.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:05.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:05.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.785 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:05.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:05.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:05.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:05.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.793 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 66.269301444999996, 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:05.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:05.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:05.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:05.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.800 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.800 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:05.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.928237549000002, 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.801 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.803 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:05.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:05.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:05.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:05.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:05.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:05.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.919411268000005, 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.814 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:05.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:05.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.918789595999996, 2023-12-06T09:23:05.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:05.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:05.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.910843115000006, 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:05.827 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:05.831 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.910369097, 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.909914741999998, 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.836 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:05.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:05.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.525922491000003, 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.840 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:05.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:05.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:05.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:05.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:05.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.847 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.140773085999996, 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.852 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.128497775, 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:05.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:05.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:05.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:05.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:05.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 65.127012196999999, 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.860 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.861 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:05.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:05.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.220475039, 2023-12-06T09:23:05.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:05.866 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.866 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.866 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:05.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:05.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:05.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:05.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:05.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.878 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.215342735999997, 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.879 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:05.880 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:05.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:05.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.214516705999998, 2023-12-06T09:23:05.883 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:05.884 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.885 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.887 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:05.887 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:05.888 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.211947244999998, 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.890 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.891 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.892 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.893 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.211201774999999, 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.894 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.895 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.896 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.210395704, 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:05.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:05.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 57.199291252000002, 2023-12-06T09:23:05.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:05.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:05.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:05.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:05.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:05.914 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:05.914 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:05.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:05.916 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:05.929+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:06.016 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:06.031+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:06.085 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:06.085 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:06.085 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:06.085 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:06.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:06.119 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:06.131+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:06.218 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:06.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:06.221 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:06.222 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:06.222 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:06.222 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:06.222 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:06.231 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:06.318 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:06.332+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:06.401 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:06.401 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:06.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:06.401 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:06.401 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:06.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:06.419 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:06.433+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:06.520 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:06.535+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:06.538 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:06.539 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:06.539 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:06.539 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:06.547 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:06.622 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:06.635+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:06.686 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:06.686 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:06.686 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:06.686 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:06.686 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:06.695 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:06.723 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:06.737+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:06.823 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:06.838+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:06.864 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 51.273634127000001, 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:06.865 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:06.868 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:06.869 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:06.869 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:06.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:06.871 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:06.871 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:06.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:06.925 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:06.938+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:07.015 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:07.015 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:07.015 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:07.015 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:07.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:07.026 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:07.040+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:07.128 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:07.142+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:07.154 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:07.154 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:07.154 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:07.154 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:07.154 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:07.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:07.228 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:07.242+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:07.297 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:07.297 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:07.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:07.297 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:07.297 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:07.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:07.329 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:07.343+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 30, 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:21:08.071156+0000", 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:22:50.836472+0000", 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: "last_up_change": "2023-12-06T09:21:55.261571+0000", 2023-12-06T09:23:07.368 INFO:teuthology.orchestra.run.smithi088.stdout: "last_in_change": "2023-12-06T09:21:13.597313+0000", 2023-12-06T09:23:07.369 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T09:23:07.369 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_num": 5806080, 2023-12-06T09:23:07.375 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_set": [ 2023-12-06T09:23:07.375 INFO:teuthology.orchestra.run.smithi088.stdout: "nodeep-scrub", 2023-12-06T09:23:07.375 INFO:teuthology.orchestra.run.smithi088.stdout: "noscrub", 2023-12-06T09:23:07.375 INFO:teuthology.orchestra.run.smithi088.stdout: "pglog_hardlimit", 2023-12-06T09:23:07.375 INFO:teuthology.orchestra.run.smithi088.stdout: "purged_snapdirs", 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_deletes", 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "sortbitwise" 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_version": 6, 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_snapshot": "", 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_max": 2, 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "max_osd": 8, 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "require_min_compat_client": "luminous", 2023-12-06T09:23:07.376 INFO:teuthology.orchestra.run.smithi088.stdout: "min_compat_client": "luminous", 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "require_osd_release": "reef", 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "allow_crimson": false, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "pools": [ 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 1, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": ".mgr", 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:21:54.386974+0000", 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:23:07.377 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:23:07.378 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:23:07.378 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:23:07.378 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 1, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 1, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 1, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 1, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 1, 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:23:07.381 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "18", 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:23:07.382 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:23:07.383 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:23:07.384 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:23:07.385 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:23:07.389 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:23:07.389 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:23:07.389 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "options": { 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_max": 32, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_min": 1 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": {} 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 8, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 8, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.25, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 2, 2023-12-06T09:23:07.390 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 2, 2023-12-06T09:23:07.391 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 0.89483833312988281, 2023-12-06T09:23:07.391 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 0.88999998569488525, 2023-12-06T09:23:07.391 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2023-12-06T09:23:07.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 2, 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": "rbd", 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:22:01.832571+0000", 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:23:07.393 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:23:07.394 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 8, 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 8, 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 8, 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 8, 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 8, 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:23:07.397 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "21", 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:23:07.398 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:23:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:23:07.400 INFO:teuthology.orchestra.run.smithi088.stdout: "options": {}, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd": {} 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 1.9900000095367432, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 1.9900000095367432, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 1.75, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 1.75, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 0.89483833312988281, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 0.88999998569488525, 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2023-12-06T09:23:07.401 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.402 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:07.411 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:23:07.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.411 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:23:07.411 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "db81b709-a318-4b43-beb2-bf3687f365e1", 2023-12-06T09:23:07.413 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.414 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6805", 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.415 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6806", 2023-12-06T09:23:07.418 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:07.418 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.418 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.419 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.419 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6808", 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.421 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6807", 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6805/2388339328", 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6806/2388339328", 2023-12-06T09:23:07.423 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6808/2388339328", 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6807/2388339328", 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd7b0482-e237-424b-8eff-8db905713049", 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.424 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 15, 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6801", 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6802", 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.426 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6804", 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.427 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6803", 2023-12-06T09:23:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.428 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.434 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6801/3646708602", 2023-12-06T09:23:07.434 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6802/3646708602", 2023-12-06T09:23:07.434 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6804/3646708602", 2023-12-06T09:23:07.434 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6803/3646708602", 2023-12-06T09:23:07.434 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.434 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e5a571be-6b8e-4c6f-bd3e-2af30c112285", 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:23:07.435 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6810", 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6813", 2023-12-06T09:23:07.436 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6816", 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6814", 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6810/3714168299", 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6813/3714168299", 2023-12-06T09:23:07.438 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6816/3714168299", 2023-12-06T09:23:07.446 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6814/3714168299", 2023-12-06T09:23:07.446 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.446 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "d7abe19a-cc3d-411f-8533-8592e1379735", 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 0.1587066650390625, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.447 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.448 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6809", 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6811", 2023-12-06T09:23:07.449 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6815", 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6812", 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6809/1828251745", 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6811/1828251745", 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1828251745", 2023-12-06T09:23:07.451 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6812/1828251745", 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b9517160-d081-44f9-aea4-6d2529e02114", 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 13, 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6800", 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.453 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6801", 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6803", 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.454 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6802", 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.455 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6800/3720978101", 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6801/3720978101", 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6803/3720978101", 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6802/3720978101", 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e", 2023-12-06T09:23:07.456 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.457 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.458 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.458 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6804", 2023-12-06T09:23:07.460 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:07.460 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.461 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.461 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.461 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6805", 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6807", 2023-12-06T09:23:07.462 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6806", 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6804/1203998358", 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6805/1203998358", 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1203998358", 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6806/1203998358", 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e7afeca8-1363-4e9b-96f6-7fab6113d9e0", 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.464 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.465 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6808", 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6809", 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.466 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.467 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6811", 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6810", 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6808/4151455937", 2023-12-06T09:23:07.474 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6809/4151455937", 2023-12-06T09:23:07.475 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6811/4151455937", 2023-12-06T09:23:07.475 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6810/4151455937", 2023-12-06T09:23:07.475 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.475 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.475 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.475 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd4b3971-3adb-44fd-b42b-b6660305beb3", 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 26, 2023-12-06T09:23:07.476 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6812", 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.477 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6813", 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6815", 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:07.478 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6814", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6812/4165054852", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6813/4165054852", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6815/4165054852", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6814/4165054852", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:07.479 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_xinfo": [ 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.229602+0000", 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.480 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.158659+0000", 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.481 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.969728+0000", 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.234010+0000", 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.482 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:50.601730+0000", 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:23:07.483 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.304649+0000", 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.484 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:23:07.485 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.485 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.485 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.671399+0000", 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:23:07.486 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.055036+0000", 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap": [ 2023-12-06T09:23:07.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "pgid": "2.7", 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: 7, 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: 0 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_items": [], 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_primaries": [], 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_temp": [], 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_temp": [], 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklist": {}, 2023-12-06T09:23:07.488 INFO:teuthology.orchestra.run.smithi088.stdout: "range_blocklist": {}, 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profiles": { 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "default": { 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "crush-failure-domain": "osd", 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "k": "2", 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "m": "1", 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin": "jerasure", 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "technique": "reed_sol_van" 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps_queue": [], 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "new_removed_snaps": [], 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "new_purged_snaps": [], 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_node_flags": {}, 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "device_class_flags": {}, 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": { 2023-12-06T09:23:07.489 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_enabled": false, 2023-12-06T09:23:07.490 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_bucket_count": 0, 2023-12-06T09:23:07.490 INFO:teuthology.orchestra.run.smithi088.stdout: "degraded_stretch_mode": 0, 2023-12-06T09:23:07.490 INFO:teuthology.orchestra.run.smithi088.stdout: "recovering_stretch_mode": 0, 2023-12-06T09:23:07.490 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_bucket": 0 2023-12-06T09:23:07.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:07.490 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:07.493 INFO:tasks.thrashosds.thrasher:j is {'epoch': 30, 'fsid': '0c48cca3-dcc0-44bd-8eed-37fea2138550', 'created': '2023-12-06T09:21:08.071156+0000', 'modified': '2023-12-06T09:22:50.836472+0000', 'last_up_change': '2023-12-06T09:21:55.261571+0000', 'last_in_change': '2023-12-06T09:21:13.597313+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T09:21:54.386974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8948383331298828, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T09:22:01.832571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8948383331298828, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'db81b709-a318-4b43-beb2-bf3687f365e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6805', 'nonce': 2388339328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6806', 'nonce': 2388339328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6808', 'nonce': 2388339328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6807', 'nonce': 2388339328}]}, 'public_addr': '172.21.15.88:6805/2388339328', 'cluster_addr': '172.21.15.88:6806/2388339328', 'heartbeat_back_addr': '172.21.15.88:6808/2388339328', 'heartbeat_front_addr': '172.21.15.88:6807/2388339328', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'fd7b0482-e237-424b-8eff-8db905713049', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6801', 'nonce': 3646708602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6802', 'nonce': 3646708602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6804', 'nonce': 3646708602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6803', 'nonce': 3646708602}]}, 'public_addr': '172.21.15.88:6801/3646708602', 'cluster_addr': '172.21.15.88:6802/3646708602', 'heartbeat_back_addr': '172.21.15.88:6804/3646708602', 'heartbeat_front_addr': '172.21.15.88:6803/3646708602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e5a571be-6b8e-4c6f-bd3e-2af30c112285', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6810', 'nonce': 3714168299}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6813', 'nonce': 3714168299}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6816', 'nonce': 3714168299}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6814', 'nonce': 3714168299}]}, 'public_addr': '172.21.15.88:6810/3714168299', 'cluster_addr': '172.21.15.88:6813/3714168299', 'heartbeat_back_addr': '172.21.15.88:6816/3714168299', 'heartbeat_front_addr': '172.21.15.88:6814/3714168299', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd7abe19a-cc3d-411f-8533-8592e1379735', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1587066650390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6809', 'nonce': 1828251745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6811', 'nonce': 1828251745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6815', 'nonce': 1828251745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6812', 'nonce': 1828251745}]}, 'public_addr': '172.21.15.88:6809/1828251745', 'cluster_addr': '172.21.15.88:6811/1828251745', 'heartbeat_back_addr': '172.21.15.88:6815/1828251745', 'heartbeat_front_addr': '172.21.15.88:6812/1828251745', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b9517160-d081-44f9-aea4-6d2529e02114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 3720978101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6801', 'nonce': 3720978101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6803', 'nonce': 3720978101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 3720978101}]}, 'public_addr': '172.21.15.132:6800/3720978101', 'cluster_addr': '172.21.15.132:6801/3720978101', 'heartbeat_back_addr': '172.21.15.132:6803/3720978101', 'heartbeat_front_addr': '172.21.15.132:6802/3720978101', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1203998358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6805', 'nonce': 1203998358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6807', 'nonce': 1203998358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1203998358}]}, 'public_addr': '172.21.15.132:6804/1203998358', 'cluster_addr': '172.21.15.132:6805/1203998358', 'heartbeat_back_addr': '172.21.15.132:6807/1203998358', 'heartbeat_front_addr': '172.21.15.132:6806/1203998358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7afeca8-1363-4e9b-96f6-7fab6113d9e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 4151455937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6809', 'nonce': 4151455937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6811', 'nonce': 4151455937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 4151455937}]}, 'public_addr': '172.21.15.132:6808/4151455937', 'cluster_addr': '172.21.15.132:6809/4151455937', 'heartbeat_back_addr': '172.21.15.132:6811/4151455937', 'heartbeat_front_addr': '172.21.15.132:6810/4151455937', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fd4b3971-3adb-44fd-b42b-b6660305beb3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 4165054852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6813', 'nonce': 4165054852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6815', 'nonce': 4165054852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 4165054852}]}, 'public_addr': '172.21.15.132:6812/4165054852', 'cluster_addr': '172.21.15.132:6813/4165054852', 'heartbeat_back_addr': '172.21.15.132:6815/4165054852', 'heartbeat_front_addr': '172.21.15.132:6814/4165054852', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.229602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.158659+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.969728+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.234010+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:50.601730+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.304649+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.671399+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.055036+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.7', 'osds': [7, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T09:23:07.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:23:07.496 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:07.496 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:07.496 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:07.496 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:07.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:07.497 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:07.497 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:07.498 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:07.506+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:07.558 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:07.558 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:07.592 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:07.606+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:07.649 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:07.671 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:07.671 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:07.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:07.729 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:07.740+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:07.749 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:07.797 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:07.811+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:07.814 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:07.814 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:07.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:07.836 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:07.837 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:07.898 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:07.912+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:07.926 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:07.927 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:07.967 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:07.967 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:07.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:08.022 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:08.035+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:08.028 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:08.029 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:08.100 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:08.114+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:08.109 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:23:08.110 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:23:08.117 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:08.117 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:08.127 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-12-06T09:23:07.932657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218272,"kb_used_data":2088,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756448,"statfs":{"total":773094113280,"available":772870602752,"internally_reserved":0,"allocated":2138112,"data_stored":1595672,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:50.566278+0000","last_change":"2023-12-06T09:22:45.154289+0000","last_active":"2023-12-06T09:22:50.566278+0000","last_peered":"2023-12-06T09:22:50.566278+0000","last_clean":"2023-12-06T09:22:50.566278+0000","last_became_active":"2023-12-06T09:22:45.140879+0000","last_became_peered":"2023-12-06T09:22:45.140879+0000","last_unstale":"2023-12-06T09:22:50.566278+0000","last_undegraded":"2023-12-06T09:22:50.566278+0000","last_fullsized":"2023-12-06T09:22:50.566278+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:29:31.043524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:50.561366+0000","last_change":"2023-12-06T09:22:04.134391+0000","last_active":"2023-12-06T09:22:50.561366+0000","last_peered":"2023-12-06T09:22:50.561366+0000","last_clean":"2023-12-06T09:22:50.561366+0000","last_became_active":"2023-12-06T09:22:04.102548+0000","last_became_peered":"2023-12-06T09:22:04.102548+0000","last_unstale":"2023-12-06T09:22:50.561366+0000","last_undegraded":"2023-12-06T09:22:50.561366+0000","last_fullsized":"2023-12-06T09:22:50.561366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:11:07.356997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:49.706660+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:22:49.706660+0000","last_peered":"2023-12-06T09:22:49.706660+0000","last_clean":"2023-12-06T09:22:49.706660+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:22:49.706660+0000","last_undegraded":"2023-12-06T09:22:49.706660+0000","last_fullsized":"2023-12-06T09:22:49.706660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:49.428860+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:22:49.428860+0000","last_peered":"2023-12-06T09:22:49.428860+0000","last_clean":"2023-12-06T09:22:49.428860+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:22:49.428860+0000","last_undegraded":"2023-12-06T09:22:49.428860+0000","last_fullsized":"2023-12-06T09:22:49.428860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3,"reported_epoch":30,"state":"peering","last_fresh":"2023-12-06T09:22:51.235769+0000","last_change":"2023-12-06T09:22:51.235769+0000","last_active":"2023-12-06T09:22:02.068230+0000","last_peered":"2023-12-06T09:22:02.068230+0000","last_clean":"2023-12-06T09:22:02.068230+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-06T09:22:51.235769+0000","last_undegraded":"2023-12-06T09:22:51.235769+0000","last_fullsized":"2023-12-06T09:22:51.235769+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:50.563438+0000","last_change":"2023-12-06T09:22:03.665768+0000","last_active":"2023-12-06T09:22:50.563438+0000","last_peered":"2023-12-06T09:22:50.563438+0000","last_clean":"2023-12-06T09:22:50.563438+0000","last_became_active":"2023-12-06T09:22:03.638099+0000","last_became_peered":"2023-12-06T09:22:03.638099+0000","last_unstale":"2023-12-06T09:22:50.563438+0000","last_undegraded":"2023-12-06T09:22:50.563438+0000","last_fullsized":"2023-12-06T09:22:50.563438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:58:46.014124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:49.698405+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:22:49.698405+0000","last_peered":"2023-12-06T09:22:49.698405+0000","last_clean":"2023-12-06T09:22:49.698405+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:22:49.698405+0000","last_undegraded":"2023-12-06T09:22:49.698405+0000","last_fullsized":"2023-12-06T09:22:49.698405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3,"reported_epoch":30,"state":"peering","last_fresh":"2023-12-06T09:22:51.558581+0000","last_change":"2023-12-06T09:22:51.558581+0000","last_active":"2023-12-06T09:22:02.068230+0000","last_peered":"2023-12-06T09:22:02.068230+0000","last_clean":"2023-12-06T09:22:02.068230+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-06T09:22:51.558581+0000","last_undegraded":"2023-12-06T09:22:51.558581+0000","last_fullsized":"2023-12-06T09:22:51.558581+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":126,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-06T09:22:50.092277+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:22:50.092277+0000","last_peered":"2023-12-06T09:22:50.092277+0000","last_clean":"2023-12-06T09:22:50.092277+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:22:50.092277+0000","last_undegraded":"2023-12-06T09:22:50.092277+0000","last_fullsized":"2023-12-06T09:22:50.092277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":17.015000000000001,"15min":17.015000000000001},"min":{"1min":5.3929999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"max":{"1min":49.691000000000003,"5min":49.691000000000003,"15min":49.691000000000003},"last":7.3369999999999997},{"interface":"front","average":{"1min":17.239999999999998,"5min":17.239999999999998,"15min":17.239999999999998},"min":{"1min":6.266,"5min":6.266,"15min":6.266},"max":{"1min":51.530000000000001,"5min":51.530000000000001,"15min":51.530000000000001},"last":4.0709999999999997}]},{"osd":1,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":12.565,"5min":12.565,"15min":12.565},"min":{"1min":5.9649999999999999,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":38.488999999999997,"5min":38.488999999999997,"15min":38.488999999999997},"last":13.73},{"interface":"front","average":{"1min":15.016999999999999,"5min":15.016999999999999,"15min":15.016999999999999},"min":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"max":{"1min":35.378999999999998,"5min":35.378999999999998,"15min":35.378999999999998},"last":14.840999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":15.605,"5min":15.605,"15min":15.605},"min":{"1min":7.2000000000000002,"5min":7.2000000000000002,"15min":7.2000000000000002},"max":{"1min":42.523000000000003,"5min":42.523000000000003,"15min":42.523000000000003},"last":10.432},{"interface":"front","average":{"1min":18.297999999999998,"5min":18.297999999999998,"15min":18.297999999999998},"min":{"1min":9.5459999999999994,"5min":9.5459999999999994,"15min":9.5459999999999994},"max":{"1min":39.066000000000003,"5min":39.066000000000003,"15min":39.066000000000003},"last":9.6430000000000007}]},{"osd":4,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":15.315,"5min":15.315,"15min":15.315},"min":{"1min":8.9199999999999999,"5min":8.9199999999999999,"15min":8.9199999999999999},"max":{"1min":25.123000000000001,"5min":25.123000000000001,"15min":25.123000000000001},"last":8.3059999999999992},{"interface":"front","average":{"1min":15.606,"5min":15.606,"15min":15.606},"min":{"1min":7.7880000000000003,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":25.466000000000001,"5min":25.466000000000001,"15min":25.466000000000001},"last":11.753}]},{"osd":5,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":27.913,"5min":27.913,"15min":27.913},"min":{"1min":8.298,"5min":8.298,"15min":8.298},"max":{"1min":190.821,"5min":190.821,"15min":190.821},"last":11.073},{"interface":"front","average":{"1min":27.640999999999998,"5min":27.640999999999998,"15min":27.640999999999998},"min":{"1min":8.5969999999999995,"5min":8.5969999999999995,"15min":8.5969999999999995},"max":{"1min":191.86199999999999,"5min":191.86199999999999,"15min":191.86199999999999},"last":16.841999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":27.119,"5min":27.119,"15min":27.119},"min":{"1min":12.289,"5min":12.289,"15min":12.289},"max":{"1min":166.45400000000001,"5min":166.45400000000001,"15min":166.45400000000001},"last":15.515000000000001},{"interface":"front","average":{"1min":27.024999999999999,"5min":27.024999999999999,"15min":27.024999999999999},"min":{"1min":11.528,"5min":11.528,"15min":11.528},"max":{"1min":117.58499999999999,"5min":117.58499999999999,"15min":117.58499999999999},"last":16.143000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":18.994,"5min":18.994,"15min":18.994},"min":{"1min":6.6619999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":32.558,"5min":32.558,"15min":32.558},"last":12.523999999999999},{"interface":"front","average":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":19.864000000000001},"min":{"1min":12.651,"5min":12.651,"15min":12.651},"max":{"1min":27.971,"5min":27.971,"15min":27.971},"last":18.181999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":99999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":7.7359999999999998,"5min":7.7359999999999998,"15min":7.7359999999999998},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":18.952000000000002,"5min":18.952000000000002,"15min":18.952000000000002},"last":7.0110000000000001},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.356999999999999,"15min":12.356999999999999},"min":{"1min":7.0049999999999999,"5min":7.0049999999999999,"15min":7.0049999999999999},"max":{"1min":19.617999999999999,"5min":19.617999999999999,"15min":19.617999999999999},"last":7.415}]},{"osd":1,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":15.377000000000001,"5min":15.377000000000001,"15min":15.377000000000001},"min":{"1min":7.319,"5min":7.319,"15min":7.319},"max":{"1min":52.530000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":9.7089999999999996},{"interface":"front","average":{"1min":18.968,"5min":18.968,"15min":18.968},"min":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":54.338000000000001,"5min":54.338000000000001,"15min":54.338000000000001},"last":16.742000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":10.951000000000001,"5min":10.951000000000001,"15min":10.951000000000001},"min":{"1min":6.5129999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"max":{"1min":27.295999999999999,"5min":27.295999999999999,"15min":27.295999999999999},"last":8.1319999999999997},{"interface":"front","average":{"1min":13.555999999999999,"5min":13.555999999999999,"15min":13.555999999999999},"min":{"1min":6.0460000000000003,"5min":6.0460000000000003,"15min":6.0460000000000003},"max":{"1min":21.73,"5min":21.73,"15min":21.73},"last":12.724}]},{"osd":4,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":15.212999999999999,"5min":15.212999999999999,"15min":15.212999999999999},"min":{"1min":7.7009999999999996,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":32.063000000000002,"5min":32.063000000000002,"15min":32.063000000000002},"last":8.8719999999999999},{"interface":"front","average":{"1min":17.879000000000001,"5min":17.879000000000001,"15min":17.879000000000001},"min":{"1min":9.2919999999999998,"5min":9.2919999999999998,"15min":9.2919999999999998},"max":{"1min":36.337000000000003,"5min":36.337000000000003,"15min":36.337000000000003},"last":18.802}]},{"osd":5,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"min":{"1min":8.4900000000000002,"5min":8.4900000000000002,"15min":8.4900000000000002},"max":{"1min":56.491,"5min":56.491,"15min":56.491},"last":11.789999999999999},{"interface":"front","average":{"1min":20.263000000000002,"5min":20.263000000000002,"15min":20.263000000000002},"min":{"1min":9.3719999999999999,"5min":9.3719999999999999,"15min":9.3719999999999999},"max":{"1min":75.540999999999997,"5min":75.540999999999997,"15min":75.540999999999997},"last":10.99}]},{"osd":6,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":18.440999999999999,"5min":18.440999999999999,"15min":18.440999999999999},"min":{"1min":11.493,"5min":11.493,"15min":11.493},"max":{"1min":30.213999999999999,"5min":30.213999999999999,"15min":30.213999999999999},"last":17.989000000000001},{"interface":"front","average":{"1min":14.678000000000001,"5min":14.678000000000001,"15min":14.678000000000001},"min":{"1min":7.6100000000000003,"5min":7.6100000000000003,"15min":7.6100000000000003},"max":{"1min":25.847999999999999,"5min":25.847999999999999,"15min":25.847999999999999},"last":15.052}]},{"osd":7,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":20.239000000000001,"5min":20.239000000000001,"15min":20.239000000000001},"min":{"1min":13.896000000000001,"5min":13.896000000000001,"15min":13.896000000000001},"max":{"1min":35.094000000000001,"5min":35.094000000000001,"15min":35.094000000000001},"last":19.724},{"interface":"front","average":{"1min":18.385000000000002,"5min":18.385000000000002,"15min":18.385000000000002},"min":{"1min":10.811,"5min":10.811,"15min":10.811},"max":{"1min":27.387,"5min":27.387,"15min":27.387},"last":15.973000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509458,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":11.225,"15min":11.225},"min":{"1min":5.0069999999999997,"5min":5.0069999999999997,"15min":5.0069999999999997},"max":{"1min":26.649000000000001,"5min":26.649000000000001,"15min":26.649000000000001},"last":4.7770000000000001},{"interface":"front","average":{"1min":11.33,"5min":11.33,"15min":11.33},"min":{"1min":6.548,"5min":6.548,"15min":6.548},"max":{"1min":21.123999999999999,"5min":21.123999999999999,"15min":21.123999999999999},"last":10.242000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":13.086,"5min":13.086,"15min":13.086},"min":{"1min":6.5570000000000004,"5min":6.5570000000000004,"15min":6.5570000000000004},"max":{"1min":24.428999999999998,"5min":24.428999999999998,"15min":24.428999999999998},"last":8.5050000000000008},{"interface":"front","average":{"1min":14.44,"5min":14.44,"15min":14.44},"min":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"max":{"1min":21.867000000000001,"5min":21.867000000000001,"15min":21.867000000000001},"last":11.673999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":23.254999999999999,"5min":23.254999999999999,"15min":23.254999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":205.096,"5min":205.096,"15min":205.096},"last":14.189},{"interface":"front","average":{"1min":24.513999999999999,"5min":24.513999999999999,"15min":24.513999999999999},"min":{"1min":8.5660000000000007,"5min":8.5660000000000007,"15min":8.5660000000000007},"max":{"1min":182.74000000000001,"5min":182.74000000000001,"15min":182.74000000000001},"last":9.5809999999999995}]},{"osd":4,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":14.353999999999999,"5min":14.353999999999999,"15min":14.353999999999999},"min":{"1min":7.5369999999999999,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":25.468,"5min":25.468,"15min":25.468},"last":19.198},{"interface":"front","average":{"1min":16.768999999999998,"5min":16.768999999999998,"15min":16.768999999999998},"min":{"1min":9.6310000000000002,"5min":9.6310000000000002,"15min":9.6310000000000002},"max":{"1min":33.637999999999998,"5min":33.637999999999998,"15min":33.637999999999998},"last":16.254999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":15.068,"5min":15.068,"15min":15.068},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":28.081,"5min":28.081,"15min":28.081},"last":10.955},{"interface":"front","average":{"1min":17.021999999999998,"5min":17.021999999999998,"15min":17.021999999999998},"min":{"1min":10.132999999999999,"5min":10.132999999999999,"15min":10.132999999999999},"max":{"1min":30.795000000000002,"5min":30.795000000000002,"15min":30.795000000000002},"last":18.274999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":21.085999999999999,"5min":21.085999999999999,"15min":21.085999999999999},"min":{"1min":7.5640000000000001,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":78.338999999999999,"5min":78.338999999999999,"15min":78.338999999999999},"last":15.358000000000001},{"interface":"front","average":{"1min":21.478999999999999,"5min":21.478999999999999,"15min":21.478999999999999},"min":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":9.4190000000000005},"max":{"1min":80.850999999999999,"5min":80.850999999999999,"15min":80.850999999999999},"last":17.004999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":20.716000000000001,"5min":20.716000000000001,"15min":20.716000000000001},"min":{"1min":11.795,"5min":11.795,"15min":11.795},"max":{"1min":46.590000000000003,"5min":46.590000000000003,"15min":46.590000000000003},"last":12.426},{"interface":"front","average":{"1min":21.113,"5min":21.113,"15min":21.113},"min":{"1min":12.242000000000001,"5min":12.242000000000001,"15min":12.242000000000001},"max":{"1min":50.432000000000002,"5min":50.432000000000002,"15min":50.432000000000002},"last":19.831}]}]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":99999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":10.112,"5min":10.112,"15min":10.112},"min":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":25.277999999999999,"5min":25.277999999999999,"15min":25.277999999999999},"last":5.5869999999999997},{"interface":"front","average":{"1min":10.464,"5min":10.464,"15min":10.464},"min":{"1min":5.117,"5min":5.117,"15min":5.117},"max":{"1min":28.972000000000001,"5min":28.972000000000001,"15min":28.972000000000001},"last":10.648999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":16.478999999999999,"5min":16.478999999999999,"15min":16.478999999999999},"min":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":5.6849999999999996},"max":{"1min":57.503,"5min":57.503,"15min":57.503},"last":9.9480000000000004},{"interface":"front","average":{"1min":17.050999999999998,"5min":17.050999999999998,"15min":17.050999999999998},"min":{"1min":7.3070000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":47.421999999999997,"5min":47.421999999999997,"15min":47.421999999999997},"last":14.781000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":16.943000000000001,"5min":16.943000000000001,"15min":16.943000000000001},"min":{"1min":6.9189999999999996,"5min":6.9189999999999996,"15min":6.9189999999999996},"max":{"1min":41.762999999999998,"5min":41.762999999999998,"15min":41.762999999999998},"last":15.48},{"interface":"front","average":{"1min":12.894,"5min":12.894,"15min":12.894},"min":{"1min":6.3070000000000004,"5min":6.3070000000000004,"15min":6.3070000000000004},"max":{"1min":42.520000000000003,"5min":42.520000000000003,"15min":42.520000000000003},"last":9.4819999999999993}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":6.6520000000000001,"5min":6.6520000000000001,"15min":6.6520000000000001},"max":{"1min":33.357999999999997,"5min":33.357999999999997,"15min":33.357999999999997},"last":12.032999999999999},{"interface":"front","average":{"1min":16.074999999999999,"5min":16.074999999999999,"15min":16.074999999999999},"min":{"1min":5.7949999999999999,"5min":5.7949999999999999,"15min":5.7949999999999999},"max":{"1min":36.402999999999999,"5min":36.402999999999999,"15min":36.402999999999999},"last":12.701000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":15.061,"5min":15.061,"15min":15.061},"min":{"1min":6.9160000000000004,"5min":6.9160000000000004,"15min":6.9160000000000004},"max":{"1min":32.531999999999996,"5min":32.531999999999996,"15min":32.531999999999996},"last":16.311},{"interface":"front","average":{"1min":16.821999999999999,"5min":16.821999999999999,"15min":16.821999999999999},"min":{"1min":8.0370000000000008,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":39.546999999999997,"5min":39.546999999999997,"15min":39.546999999999997},"last":13.351000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":21.998999999999999,"5min":21.998999999999999,"15min":21.998999999999999},"min":{"1min":9.218,"5min":9.218,"15min":9.218},"max":{"1min":77.706999999999994,"5min":77.706999999999994,"15min":77.706999999999994},"last":17.125},{"interface":"front","average":{"1min":22.5,"5min":22.5,"15min":22.5},"min":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"max":{"1min":69.787999999999997,"5min":69.787999999999997,"15min":69.787999999999997},"last":17.82}]},{"osd":6,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":20.309000000000001,"5min":20.309000000000001,"15min":20.309000000000001},"min":{"1min":10.097,"5min":10.097,"15min":10.097},"max":{"1min":62.383000000000003,"5min":62.383000000000003,"15min":62.383000000000003},"last":14.058},{"interface":"front","average":{"1min":21.812000000000001,"5min":21.812000000000001,"15min":21.812000000000001},"min":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"max":{"1min":63.496000000000002,"5min":63.496000000000002,"15min":63.496000000000002},"last":18.559000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542160,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":99999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":12.169,"5min":12.169,"15min":12.169},"min":{"1min":4.218,"5min":4.218,"15min":4.218},"max":{"1min":45.506,"5min":45.506,"15min":45.506},"last":4.1959999999999997},{"interface":"front","average":{"1min":13.605,"5min":13.605,"15min":13.605},"min":{"1min":5.9530000000000003,"5min":5.9530000000000003,"15min":5.9530000000000003},"max":{"1min":46.450000000000003,"5min":46.450000000000003,"15min":46.450000000000003},"last":8.6080000000000005}]},{"osd":1,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":16.016999999999999,"5min":16.016999999999999,"15min":16.016999999999999},"min":{"1min":7.2140000000000004,"5min":7.2140000000000004,"15min":7.2140000000000004},"max":{"1min":55.398000000000003,"5min":55.398000000000003,"15min":55.398000000000003},"last":37.103999999999999},{"interface":"front","average":{"1min":15.51,"5min":15.51,"15min":15.51},"min":{"1min":7.8559999999999999,"5min":7.8559999999999999,"15min":7.8559999999999999},"max":{"1min":41.811999999999998,"5min":41.811999999999998,"15min":41.811999999999998},"last":38.137}]},{"osd":2,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":22.890000000000001,"5min":22.890000000000001,"15min":22.890000000000001},"min":{"1min":5.1420000000000003,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":208.93799999999999,"5min":208.93799999999999,"15min":208.93799999999999},"last":9.8369999999999997},{"interface":"front","average":{"1min":27.946000000000002,"5min":27.946000000000002,"15min":27.946000000000002},"min":{"1min":10.807,"5min":10.807,"15min":10.807},"max":{"1min":208.57400000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":11.807}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":18.024000000000001,"5min":18.024000000000001,"15min":18.024000000000001},"min":{"1min":7.032,"5min":7.032,"15min":7.032},"max":{"1min":43.276000000000003,"5min":43.276000000000003,"15min":43.276000000000003},"last":15.199},{"interface":"front","average":{"1min":15.728999999999999,"5min":15.728999999999999,"15min":15.728999999999999},"min":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"max":{"1min":41.493000000000002,"5min":41.493000000000002,"15min":41.493000000000002},"last":13.737}]},{"osd":4,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":25.762,"5min":25.762,"15min":25.762},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":117.10299999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":11.167},{"interface":"front","average":{"1min":28.823,"5min":28.823,"15min":28.823},"min":{"1min":9.1460000000000008,"5min":9.1460000000000008,"15min":9.1460000000000008},"max":{"1min":111.874,"5min":111.874,"15min":111.874},"last":18.625}]},{"osd":5,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":18.038,"5min":18.038,"15min":18.038},"min":{"1min":8.0410000000000004,"5min":8.0410000000000004,"15min":8.0410000000000004},"max":{"1min":39.886000000000003,"5min":39.886000000000003,"15min":39.886000000000003},"last":14.475},{"interface":"front","average":{"1min":23.875,"5min":23.875,"15min":23.875},"min":{"1min":13.242000000000001,"5min":13.242000000000001,"15min":13.242000000000001},"max":{"1min":60.530999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":12.542}]},{"osd":7,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":25.402999999999999,"5min":25.402999999999999,"15min":25.402999999999999},"min":{"1min":9.6839999999999993,"5min":9.6839999999999993,"15min":9.6839999999999993},"max":{"1min":87.590999999999994,"5min":87.590999999999994,"15min":87.590999999999994},"last":16.681999999999999},{"interface":"front","average":{"1min":21.391999999999999,"5min":21.391999999999999,"15min":21.391999999999999},"min":{"1min":9.6929999999999996,"5min":9.6929999999999996,"15min":9.6929999999999996},"max":{"1min":50.737000000000002,"5min":50.737000000000002,"15min":50.737000000000002},"last":17.672999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":497839,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.371,"15min":14.371},"min":{"1min":5.4210000000000003,"5min":5.4210000000000003,"15min":5.4210000000000003},"max":{"1min":45.491,"5min":45.491,"15min":45.491},"last":8.3780000000000001},{"interface":"front","average":{"1min":13.964,"5min":13.964,"15min":13.964},"min":{"1min":4.2859999999999996,"5min":4.2859999999999996,"15min":4.2859999999999996},"max":{"1min":49.991999999999997,"5min":49.991999999999997,"15min":49.991999999999997},"last":5.29}]},{"osd":1,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":25.047000000000001,"5min":25.047000000000001,"15min":25.047000000000001},"min":{"1min":4.6710000000000003,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":220.13900000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":7.7050000000000001},{"interface":"front","average":{"1min":35.741999999999997,"5min":35.741999999999997,"15min":35.741999999999997},"min":{"1min":7.476,"5min":7.476,"15min":7.476},"max":{"1min":344.79599999999999,"5min":344.79599999999999,"15min":344.79599999999999},"last":16.547999999999998}]},{"osd":2,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":13.627000000000001,"5min":13.627000000000001,"15min":13.627000000000001},"min":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":53.235999999999997,"5min":53.235999999999997,"15min":53.235999999999997},"last":9.6999999999999993},{"interface":"front","average":{"1min":15.888999999999999,"5min":15.888999999999999,"15min":15.888999999999999},"min":{"1min":9.1989999999999998,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":54.747,"5min":54.747,"15min":54.747},"last":17.419}]},{"osd":3,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":13.504,"5min":13.504,"15min":13.504},"min":{"1min":6.4189999999999996,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":46.685000000000002,"5min":46.685000000000002,"15min":46.685000000000002},"last":10.526999999999999},{"interface":"front","average":{"1min":17.748999999999999,"5min":17.748999999999999,"15min":17.748999999999999},"min":{"1min":8.8670000000000009,"5min":8.8670000000000009,"15min":8.8670000000000009},"max":{"1min":51.045999999999999,"5min":51.045999999999999,"15min":51.045999999999999},"last":11.781000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":22.210999999999999,"5min":22.210999999999999,"15min":22.210999999999999},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":281.608,"5min":281.608,"15min":281.608},"last":13.464},{"interface":"front","average":{"1min":24.216000000000001,"5min":24.216000000000001,"15min":24.216000000000001},"min":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":7.7199999999999998},"max":{"1min":225.83600000000001,"5min":225.83600000000001,"15min":225.83600000000001},"last":18.256}]},{"osd":6,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":22.564,"5min":22.564,"15min":22.564},"min":{"1min":8.4109999999999996,"5min":8.4109999999999996,"15min":8.4109999999999996},"max":{"1min":198.667,"5min":198.667,"15min":198.667},"last":11.157},{"interface":"front","average":{"1min":24.366,"5min":24.366,"15min":24.366},"min":{"1min":9.1699999999999999,"5min":9.1699999999999999,"15min":9.1699999999999999},"max":{"1min":208.446,"5min":208.446,"15min":208.446},"last":12.448}]},{"osd":7,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":27.131,"5min":27.131,"15min":27.131},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":229.83500000000001,"5min":229.83500000000001,"15min":229.83500000000001},"last":19.181000000000001},{"interface":"front","average":{"1min":24.824999999999999,"5min":24.824999999999999,"15min":24.824999999999999},"min":{"1min":8.5340000000000007,"5min":8.5340000000000007,"15min":8.5340000000000007},"max":{"1min":233.59700000000001,"5min":233.59700000000001,"15min":233.59700000000001},"last":14.1}]}]},{"osd":1,"up_from":14,"seq":60129542160,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":497839,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":11.372999999999999,"5min":11.372999999999999,"15min":11.372999999999999},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":40.225999999999999,"5min":40.225999999999999,"15min":40.225999999999999},"last":7.875},{"interface":"front","average":{"1min":12.084,"5min":12.084,"15min":12.084},"min":{"1min":4.4950000000000001,"5min":4.4950000000000001,"15min":4.4950000000000001},"max":{"1min":30.405000000000001,"5min":30.405000000000001,"15min":30.405000000000001},"last":6.6639999999999997}]},{"osd":2,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":10.69,"5min":10.69,"15min":10.69},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":23.539999999999999,"5min":23.539999999999999,"15min":23.539999999999999},"last":7.3040000000000003},{"interface":"front","average":{"1min":15.456,"5min":15.456,"15min":15.456},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":45.158000000000001,"5min":45.158000000000001,"15min":45.158000000000001},"last":13.41}]},{"osd":3,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":22.263999999999999,"5min":22.263999999999999,"15min":22.263999999999999},"min":{"1min":5.8230000000000004,"5min":5.8230000000000004,"15min":5.8230000000000004},"max":{"1min":122.762,"5min":122.762,"15min":122.762},"last":10.550000000000001},{"interface":"front","average":{"1min":21.706,"5min":21.706,"15min":21.706},"min":{"1min":7.742,"5min":7.742,"15min":7.742},"max":{"1min":129.99100000000001,"5min":129.99100000000001,"15min":129.99100000000001},"last":9.9960000000000004}]},{"osd":4,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":32.768999999999998,"5min":32.768999999999998,"15min":32.768999999999998},"min":{"1min":7.944,"5min":7.944,"15min":7.944},"max":{"1min":359.46300000000002,"5min":359.46300000000002,"15min":359.46300000000002},"last":11.531000000000001},{"interface":"front","average":{"1min":31.254999999999999,"5min":31.254999999999999,"15min":31.254999999999999},"min":{"1min":8.1229999999999993,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":314.47699999999998,"5min":314.47699999999998,"15min":314.47699999999998},"last":14.662000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":28.239000000000001,"5min":28.239000000000001,"15min":28.239000000000001},"min":{"1min":9.9770000000000003,"5min":9.9770000000000003,"15min":9.9770000000000003},"max":{"1min":152.97399999999999,"5min":152.97399999999999,"15min":152.97399999999999},"last":12.816000000000001},{"interface":"front","average":{"1min":26.779,"5min":26.779,"15min":26.779},"min":{"1min":10.106,"5min":10.106,"15min":10.106},"max":{"1min":154.53100000000001,"5min":154.53100000000001,"15min":154.53100000000001},"last":18.021999999999998}]},{"osd":6,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":17.939,"5min":17.939,"15min":17.939},"min":{"1min":8.3399999999999999,"5min":8.3399999999999999,"15min":8.3399999999999999},"max":{"1min":43.045000000000002,"5min":43.045000000000002,"15min":43.045000000000002},"last":15.35},{"interface":"front","average":{"1min":21.443000000000001,"5min":21.443000000000001,"15min":21.443000000000001},"min":{"1min":11.129,"5min":11.129,"15min":11.129},"max":{"1min":41.356000000000002,"5min":41.356000000000002,"15min":41.356000000000002},"last":13.988}]},{"osd":7,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":34.292000000000002,"5min":34.292000000000002,"15min":34.292000000000002},"min":{"1min":13.757,"5min":13.757,"15min":13.757},"max":{"1min":285.81700000000001,"5min":285.81700000000001,"15min":285.81700000000001},"last":16.916},{"interface":"front","average":{"1min":33.560000000000002,"5min":33.560000000000002,"15min":33.560000000000002},"min":{"1min":9.3260000000000005,"5min":9.3260000000000005,"15min":9.3260000000000005},"max":{"1min":295.28500000000003,"5min":295.28500000000003,"15min":295.28500000000003},"last":18.623999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":99999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":14.59,"5min":14.59,"15min":14.59},"min":{"1min":4.3079999999999998,"5min":4.3079999999999998,"15min":4.3079999999999998},"max":{"1min":120.748,"5min":120.748,"15min":120.748},"last":8.9410000000000007},{"interface":"front","average":{"1min":20.132000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"min":{"1min":4.2930000000000001,"5min":4.2930000000000001,"15min":4.2930000000000001},"max":{"1min":199.268,"5min":199.268,"15min":199.268},"last":16.375}]},{"osd":1,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":20.681999999999999,"15min":20.681999999999999},"min":{"1min":7.3860000000000001,"5min":7.3860000000000001,"15min":7.3860000000000001},"max":{"1min":92.817999999999998,"5min":92.817999999999998,"15min":92.817999999999998},"last":7.7960000000000003},{"interface":"front","average":{"1min":21.934000000000001,"5min":21.934000000000001,"15min":21.934000000000001},"min":{"1min":7.1989999999999998,"5min":7.1989999999999998,"15min":7.1989999999999998},"max":{"1min":95.665999999999997,"5min":95.665999999999997,"15min":95.665999999999997},"last":10.680999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":25.824999999999999,"5min":25.824999999999999,"15min":25.824999999999999},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":208.06,"5min":208.06,"15min":208.06},"last":15.738},{"interface":"front","average":{"1min":19.771999999999998,"5min":19.771999999999998,"15min":19.771999999999998},"min":{"1min":6.915,"5min":6.915,"15min":6.915},"max":{"1min":125.047,"5min":125.047,"15min":125.047},"last":9.8119999999999994}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":20.233000000000001,"5min":20.233000000000001,"15min":20.233000000000001},"min":{"1min":5.7640000000000002,"5min":5.7640000000000002,"15min":5.7640000000000002},"max":{"1min":121.627,"5min":121.627,"15min":121.627},"last":11.941000000000001},{"interface":"front","average":{"1min":19.085999999999999,"5min":19.085999999999999,"15min":19.085999999999999},"min":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":7.7110000000000003},"max":{"1min":74.602000000000004,"5min":74.602000000000004,"15min":74.602000000000004},"last":11.297000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":7.9130000000000003,"5min":7.9130000000000003,"15min":7.9130000000000003},"max":{"1min":71.759,"5min":71.759,"15min":71.759},"last":13.881},{"interface":"front","average":{"1min":17.98,"5min":17.98,"15min":17.98},"min":{"1min":8.8450000000000006,"5min":8.8450000000000006,"15min":8.8450000000000006},"max":{"1min":70.882999999999996,"5min":70.882999999999996,"15min":70.882999999999996},"last":6.8849999999999998}]},{"osd":6,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":21.596,"5min":21.596,"15min":21.596},"min":{"1min":9.3879999999999999,"5min":9.3879999999999999,"15min":9.3879999999999999},"max":{"1min":80.668999999999997,"5min":80.668999999999997,"15min":80.668999999999997},"last":17.163},{"interface":"front","average":{"1min":23.076000000000001,"5min":23.076000000000001,"15min":23.076000000000001},"min":{"1min":11.746,"5min":11.746,"15min":11.746},"max":{"1min":72.796999999999997,"5min":72.796999999999997,"15min":72.796999999999997},"last":14.484}]},{"osd":7,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":29.366,"5min":29.366,"15min":29.366},"min":{"1min":10.706,"5min":10.706,"15min":10.706},"max":{"1min":207.19399999999999,"5min":207.19399999999999,"15min":207.19399999999999},"last":13.273},{"interface":"front","average":{"1min":29.623999999999999,"5min":29.623999999999999,"15min":29.623999999999999},"min":{"1min":11.667999999999999,"5min":11.667999999999999,"15min":11.667999999999999},"max":{"1min":208.68000000000001,"5min":208.68000000000001,"15min":208.68000000000001},"last":18.614999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:23:08.130 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [5, 0, 7, 2] 2023-12-06T09:23:08.130 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '5', '0', '7', '2'] 2023-12-06T09:23:08.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.1 5 0 7 2 2023-12-06T09:23:08.131 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:08.131 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:08.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:08.223 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:08.237+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:23:08.345 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:08.359 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:23:08.359 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T09:23:08.359 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T09:23:08.401 INFO:tasks.ceph.mon.c:Started 2023-12-06T09:23:08.401 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:23:08.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:08.405 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:08.419+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:08.445 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:08.459+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:08.446 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:08.446 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:08.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:08.506 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:08.520+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:08.607 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:08.621+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:08.630 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:08.630 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:08.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:08.709 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:08.722+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:08.809 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:08.809 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:08.809 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:08.823+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:08.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:08.917 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:08.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:09.047 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:09.055+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:09.073 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:09.073 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:09.084 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:09.105 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:09.130 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:09.143+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:09.213 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:09.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:09.248 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:09.249 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:09.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:09.314 INFO:teuthology.orchestra.run.smithi088.stderr:set 2.1 pg_upmap_items mapping to [5->0,7->2] 2023-12-06T09:23:09.340 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:09.354+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:09.417 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:09.431+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:09.426 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:09.427 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:09.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:09.518 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:09.533+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:09.607 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:09.607 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:09.616 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:09.621 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:09.635+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:09.730 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:09.744+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:09.746 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:09.746 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:09.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:09.823 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:09.837+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:09.926 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:09.926 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:09.926 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:09.940+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:09.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:10.028 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:10.042+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:10.108 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:10.108 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:10.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:10.134 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:10.135 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:10.143+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:10.230 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:10.244+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:10.288 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:10.288 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:10.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:10.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:23:10.346 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:10.357+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:10.375 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:10.375 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:10.429 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:10.429 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:10.432 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:10.446+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:10.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:10.470 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:10.471 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:10.535 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:10.547+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:10.604 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:10.605 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:10.615 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:10.615 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:10.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:10.703 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:10.736 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:10.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:10.736 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:10.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:10.811 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:10.811 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:10.830 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:10.831 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:10.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:10.855 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:10.869+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:10.918 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:10.942 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:10.955+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:10.989 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:10.989 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:10.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:11.018 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:11.018 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:11.045 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:11.059+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:11.119 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:11.119 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:11.159 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:11.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:11.175 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 55.584782304000001, 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:11.176 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:11.179 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:11.180 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:11.180 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:11.180 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:11.180 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:11.186 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:11.245 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:11.259+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:11.336 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:11.336 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:11.336 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:11.336 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:11.348 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:11.360+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:11.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:11.467 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:11.480+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:11.483 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:11.483 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:11.483 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:11.483 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:11.483 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:11.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:11.575 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:11.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:11.649 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:11.663+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:11.658 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:11.658 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:11.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:11.659 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:11.659 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:11.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:11.749 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:11.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:11.815 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:23:11.849 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:11.850 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:11.850 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:11.850 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:11.850 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:11.864+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:11.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:11.951 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:11.965+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:12.041 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:12.042 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:12.042 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:12.042 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:12.042 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:12.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:12.052 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:12.066+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:12.120 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:12.162 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:12.176+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:12.255 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:12.269+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.348176988000006, 2023-12-06T09:23:12.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.930466870999993, 2023-12-06T09:23:12.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.926282426, 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:12.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:12.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.277 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.840751339999997, 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.278 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:12.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:12.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.526774724999996, 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.525256881000004, 2023-12-06T09:23:12.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:12.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:12.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.524591505999993, 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.294 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:12.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:12.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.520331334000005, 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.300 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:12.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:12.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.519698125000005, 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.305 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.519071022999995, 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:12.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:12.315 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.430811245000001, 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:12.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:12.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:12.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:12.321 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 72.429890264999997, 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:12.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:12.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:12.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:12.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 71.677512333999999, 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:12.332 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:12.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:12.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:12.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:12.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:12.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.342 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.807477362999997, 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.343 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:12.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:12.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:12.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.350 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.806303851999999, 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:12.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.798151881000003, 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:12.356 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:12.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:12.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.362 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.787869944999997, 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:12.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:12.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:12.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:12.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.787212064999999, 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:12.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:12.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:12.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.786537285999998, 2023-12-06T09:23:12.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:12.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:12.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:12.380 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 63.754179694999998, 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.381 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:12.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:12.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:12.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:12.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:12.386 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:12.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:12.388 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:12.402+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:12.489 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:12.502+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:12.525 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:12.525 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:12.526 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:12.526 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:12.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:12.592 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:12.604+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:12.691 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:12.704+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:12.708 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:12.708 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:12.709 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:12.709 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:12.709 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:12.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:12.793 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:12.807+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:12.802 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:12.893 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:12.907+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:12.896 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:12.896 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:12.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:12.896 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:12.896 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:12.907 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:12.994 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:13.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:13.038 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:13.039 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:13.039 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:13.039 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:13.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:13.095 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:13.109+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:13.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:23:13.193 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:13.193 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:13.193 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:13.193 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:13.193 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:13.195 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:13.209+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:13.203 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:13.299 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:13.312+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:13.376 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 58.414373060999999, 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:13.386 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:13.392 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:13.401 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:13.402 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:13.403 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:13.417+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:13.483 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:13.483 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:13.505 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:13.518+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:13.529 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:13.529 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:13.529 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:13.529 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:13.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:13.571 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:13.605 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:13.620+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:13.658 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:13.707 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:13.721+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:13.758 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:13.759 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:13.759 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:13.759 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:13.769 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:13.813 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:13.827+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:13.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:23:13.852 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:13.852 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:13.900 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:13.900 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:13.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:13.910 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:13.923+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:13.953 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:13.953 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:14.010 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:14.024+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:14.050 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:14.051 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:14.079 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:14.080 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:14.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:14.115 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:14.129+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:14.212 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:14.226+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:14.216 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:14.216 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:14.227 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:14.314 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:14.327+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:14.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:14.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:14.325 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:14.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2023-12-06T09:23:14.365 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:14.365 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:14.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:14.423 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:14.437+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:14.513 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:14.513 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:14.514 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:14.528+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:14.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:14.617 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:14.630+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:14.694 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:14.694 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:14.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:14.718 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:14.732+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:14.819 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:14.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:14.876 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:14.877 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:14.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:14.905 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:23:14.928 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:14.942+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:15.022 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:15.036+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:15.022 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:23:15.025+0000 ff5f640 -1 mon.b@1(leader).osd e34 definitely_dead 0 2023-12-06T09:23:15.057 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:15.057 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:15.068 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:15.123 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:15.137+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:15.224 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:15.238+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:15.237 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:15.237 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:15.247 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:15.324 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:15.338+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:15.374 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:15.375 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:15.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:15.435 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:15.447+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:15.523 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:15.523 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:15.526 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:15.540+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:15.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:15.627 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:15.641+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:15.704 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:15.704 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:15.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:15.728 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:15.742+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:15.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:15.829 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:15.843+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:15.890 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:15.890 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:15.899 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:15.945 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:15.959+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:16.036 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:16.045+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:16.037 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:16.037 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:16.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:16.063 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:16.141 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:16.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:16.236 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:16.236 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:16.236 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:16.250+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:16.246 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:16.339 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:16.353+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:16.345 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:16.345 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:16.390 INFO:teuthology.orchestra.run.smithi088.stderr:marked out osd.0. 2023-12-06T09:23:16.440 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:16.454+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:16.442 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:16.443 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:16.567 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 79.655252688999994, 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:16.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:16.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:16.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:16.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:16.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:16.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 77.237542571999995, 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.577 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:16.582 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 77.233358127000002, 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.583 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.584 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:16.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:16.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:16.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 77.147827040999999, 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.833850425999998, 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.625 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.625 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:16.625 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.832332582000006, 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.626 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:16.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:16.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:16.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.831667206999995, 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.632 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.633 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:16.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.633 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:16.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:16.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:16.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:16.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:16.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:16.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:16.643 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.827407034999993, 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:16.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:16.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.654 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.826773825999993, 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:16.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:16.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:16.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:16.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.826146723999997, 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.663 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:16.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:16.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:16.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:16.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.737886946000003, 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:16.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.736965966, 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:16.675 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:16.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:16.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:16.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.984588035000002, 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:16.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:16.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.687 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.114553064000006, 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.688 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:16.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:16.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.113379553000001, 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:16.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:16.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:16.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:16.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.105227581999998, 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:16.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.094945645999999, 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:16.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:16.719 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.094287765999994, 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:16.725 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.093612987, 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:16.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:16.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:16.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:16.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 68.061255396000007, 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:16.732 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:16.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:16.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:16.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:16.738 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:16.740 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:16.740 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:16.590+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:16.741 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:16.741 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:16.742 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:16.742 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:16.743 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:16.743 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:16.754+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:16.791 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:16.791 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:16.841 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:16.855+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:16.872 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.409 97447== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:16.874 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:16.875 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:16.911 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:16.911 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:16.911 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:16.912 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:16.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:23:16.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:16.959 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:16.960 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:16.961 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:16.975+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:17.043 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:17.056+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:17.063 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:17.066 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:17.136 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.672 97447== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:17.142 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:17.142 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:17.142 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:17.142 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:17.142 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:17.144 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:17.158+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:17.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:17.244 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:17.258+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:17.300 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:17.300 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:17.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:17.300 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:17.300 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:17.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:17.353 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:17.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:17.447 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:17.460+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:17.459 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:17.459 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:17.459 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:17.460 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:17.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:17.555 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:17.569+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:17.643 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:17.644 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:17.644 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:17.644 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:17.644 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:17.648 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:17.662+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:17.653 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:17.749 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:17.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:17.848 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 62.885871334000001, 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:17.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:17.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:17.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:17.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:17.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:17.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:17.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:17.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:17.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:17.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:17.855 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:17.855 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:17.856 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:17.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:17.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:17.958 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:17.972+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:17.986 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:17.987 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:17.987 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:17.987 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:17.998 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:18.086 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:18.099+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:18.160 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:18.175+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:18.166 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:18.167 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:18.167 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:18.167 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:18.167 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:18.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:18.262 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:18.276+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:18.343 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:18.344 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:18.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:18.396 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:18.409+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:18.464 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:18.478+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:18.510 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:23:18.528 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:18.528 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:18.528 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:18.528 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:18.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:18.582 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:18.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:18.668 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:18.682+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:18.719 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:18.720 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:18.720 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:18.720 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:18.720 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:18.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:18.768 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:18.782+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:18.897 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:18.897 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:18.897 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 80.192673068999994, 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.498219199999994, 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:18.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:18.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.904 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:18.906 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:18.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.906 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:18.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.396390620000005, 2023-12-06T09:23:18.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:18.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.055326723999997, 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:18.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.046500442999999, 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:18.918 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:18.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.045878771000005, 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.923 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:18.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:18.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:18.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:18.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.037932290000001, 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.931 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.037458271999995, 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:18.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:18.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.939 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 79.037003917000007, 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.941 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.944 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:18.944 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:18.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:18.947 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:18.948 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 78.653011665999998, 2023-12-06T09:23:18.948 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:18.948 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.948 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.948 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:18.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.953 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 78.267862261000005, 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:18.960 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 78.255586949999994, 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.962 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.963 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 78.254101371999994, 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.969 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:18.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:18.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.347564214000002, 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.974 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:18.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.342431911000006, 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:18.978 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.341605881000007, 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.339036419999999, 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:18.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:18.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:18.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.991 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.338290950000001, 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:18.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.337484879000002, 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:18.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:18.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:19.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:19.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.006 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 70.326380427000004, 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:19.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:19.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:19.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:19.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:19.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:19.012 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.013 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:18.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:19.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:19.055 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:19.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:19.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:23:19.118 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:19.132+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:19.184 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:19.185 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:19.185 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:19.185 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:19.236 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:19.250+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:19.324 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:19.325 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:19.339+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:19.365 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:19.365 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:19.365 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:19.365 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:19.365 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:19.414 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:19.414 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:19.421 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:19.435+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:19.507 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:19.507 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:19.507 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:19.507 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.517 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:19.521 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:19.535+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:19.603 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:19.603 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:19.622 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:19.636+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:19.645 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:34.679 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:19.654 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:19.654 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:19.654 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:19.654 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.663 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:19.718 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:19.719 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:19.719 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:34.753 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:19.723 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:19.737+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:19.743 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:19.765 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:34.799 86136== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:19.768 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:43.441 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:19.772 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:43.445 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:19.799 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:43.472 94333== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:19.803 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:19.803 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:19.803 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:19.803 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:19.803 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:19.823 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:19.837+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:19.849 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:19.925 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:19.938+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:19.951 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 64.360502233000005, 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:19.952 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:19.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:19.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:19.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:19.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:19.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:19.959 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:19.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:20.027 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:20.041+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:20.095 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:20.096 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:20.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:20.132 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:20.146+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:20.232 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:20.232 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:20.233 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:20.247+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:20.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:20.333 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:20.348+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:20.373 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:20.374 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:20.383 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:20.469 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:20.481+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:20.517 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:20.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:23:20.536 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:20.550+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:20.554 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:20.555 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:20.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:20.637 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:20.651+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:20.710 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:12.247 97447== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:20.739 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:20.739 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:20.739 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:20.753+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:20.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:20.766 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:12.303 97447== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:20.840 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:20.854+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:20.904 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:20.904 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:20.913 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:20.941 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:20.955+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:21.029 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:12.566 97447== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:21.044 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:21.058+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:21.059 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:21.059 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:21.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:21.157 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:21.171+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:21.246 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:21.260+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:21.247 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:21.247 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:21.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:21.347 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:21.361+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:21.408 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:21.408 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:21.408 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2023-12-06T09:23:21.408 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T09:23:21.408 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T09:23:21.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2023-12-06T09:23:21.429 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:21.429 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:21.438 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:21.480 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:21.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:21.557 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:21.569+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:21.578 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:21.578 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T09:23:21.579 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:21.629 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:21.630 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:21.640 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:21.651 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:21.665+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:21.752 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:21.766+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:21.814 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:21.815 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:21.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:21.853 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:21.867+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:21.954 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:21.968+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:21.954 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:21.954 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:21.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:22.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:22.055 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:22.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:22.111 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:22.140 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:22.141 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:22.155 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:22.169+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:23:22.256 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:22.270+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:22.313 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:22.313 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:22.326 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:22.333 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:22.333 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:22.358 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:22.371+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:22.419 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:22.459 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:22.473+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:22.492 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:22.492 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:22.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:22.502 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:22.560 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:22.574+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:22.587 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:22.647 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:22.647 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:22.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:22.663 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:22.677+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:22.691 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:22.764 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:22.778+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:22.777 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:22.778 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:22.854 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:22.854 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:22.854 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:22.854 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:22.854 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:22.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:22.866 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:22.880+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:22.888 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:22.888 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:23.005 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:23.019+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:23.006 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:23.006 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:23.041 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:23.041 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:23.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:23.041 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:23.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.041 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:23.042 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:23.042 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 84.332720737000002, 2023-12-06T09:23:23.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:23.042 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:23.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:23.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.638266868000002, 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:23.054 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:23.059 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.536438287999999, 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:23.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:23.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:23.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:23.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.195374392000005, 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:23.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:23.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.186548110999993, 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:23.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:23.076 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.077 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.078 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.185926438999999, 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:23.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:23.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:23.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:23.083 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.177979957999995, 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:23.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:23.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.088 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.088 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:23.089 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:23.089 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.177505940000003, 2023-12-06T09:23:23.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:23.089 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:23.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:23.093 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.177051585000001, 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:23.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:23.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.099 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:23.099 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:23.099 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 82.793059334000006, 2023-12-06T09:23:23.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.103 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 82.407909928999999, 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 82.395634618000003, 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:23.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:23.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 82.394149040000002, 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.487611881999996, 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:23.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:23.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.482479579, 2023-12-06T09:23:23.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:23.126 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.481653549000001, 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:23.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:23.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:23.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:23.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.131 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:23.132 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.479084087999993, 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.478338617999995, 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:23.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.140 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:23.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:23.141 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.141 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.477532546999996, 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:23.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:23.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 74.466428094999998, 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:23.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:23.149 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:23.150 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:23.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:23.153 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:23.165+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:23.255 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:23.269+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:23.280 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:23.280 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:23.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:23.280 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:23.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:23.374 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:23.388+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:23.457 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:23.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:23.457 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:23.457 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:23.457 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:23.458 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:23.458 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:23.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:23.531 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:23.575 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:23.589+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:23.638 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:23.638 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:23.639 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:23.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:23.691 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:23.705+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:23.760 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:23.774+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:23.776 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:23.776 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:23.776 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:23.776 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:23.785 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:23.870 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:23.884+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:23.958 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:23.958 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:23.958 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:23.958 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:23.958 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:23.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:24.010 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:24.023+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:24.069 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:24.083+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:24.140 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 68.549041618000004, 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:24.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:24.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:24.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:24.144 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:24.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:24.170 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:24.184+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:24.181 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:24.194 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:23:24.194 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T09:23:24.194 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:23:24.206 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:23:24.207 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:23:24.212+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86131) UID: 0 2023-12-06T09:23:24.209 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:23:24.214+0000 9f54640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2023-12-06T09:23:24.278 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:24.292+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:24.308 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:24.308 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:24.308 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:24.309 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:24.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:24.409 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:24.419+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:24.478 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:24.492+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:24.509 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:24.509 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:24.510 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:24.510 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:24.510 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:24.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:24.574 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:24.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:24.680 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:39.714 86136== 2023-12-06T09:23:24.680 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:39.714 86136== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:23:24.691 INFO:tasks.ceph.mon.b.smithi132.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:23:24.698 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:24.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:24.724 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:24.724 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:24.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:24.777 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:24.791+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:24.877 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:24.891+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:24.924 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:24.924 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:24.924 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:24.924 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:24.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:24.980 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:24.993+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:25.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:23:25.065 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:25.066 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:25.066 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:25.066 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:25.066 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:25.075 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:25.119 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:25.133+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:25.208 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:25.222+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:25.281 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:25.295+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:25.296 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 88.37324667, 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:25.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:25.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:25.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.955536553000002, 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.951352107999995, 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.322 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:25.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.322 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:25.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:25.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:25.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:25.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.865821022000006, 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:25.336 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:25.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.551844407000004, 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.352 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:25.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.550326562999999, 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.549661188000002, 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.545401016, 2023-12-06T09:23:25.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:25.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:25.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:25.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:25.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.544767806999999, 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:25.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.544140705000004, 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:25.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:25.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:25.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.455880926999995, 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:25.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:25.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:25.396 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.454959947000006, 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.397 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:25.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:25.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.402 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 84.702582015999994, 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.832547044999998, 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:25.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.831373533999994, 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:25.413 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.418 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.823221563000004, 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:25.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:25.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:25.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:25.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:25.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:25.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.812939627000006, 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.425 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:25.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.812281747, 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:25.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:25.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:25.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:25.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:25.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:25.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.811606968000007, 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.441 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:25.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:25.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:25.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:25.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:25.446 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 76.779249376999999, 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:25.452 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:25.452 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:25.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:25.454 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:25.454 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:25.454 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:25.454 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:25.455 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:25.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:25.491 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:25.492 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:25.580 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:25.580 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:25.580 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:25.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:25.620 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:25.621 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:25.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:25.656 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:25.670+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:25.664 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:25.665 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:25.748 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:25.749 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:25.775 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:25.789+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:25.797 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:25.797 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:25.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:25.835 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:25.835 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:25.857 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:25.871+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:25.921 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:25.936 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:25.937 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:25.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:25.988 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:26.002+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:26.059 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:26.074+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:26.115 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:26.115 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:26.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:26.160 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:26.174+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:26.253 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:26.253 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:26.261 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:26.276+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:26.262 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:26.364 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:26.376+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:26.398 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:26.399 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:26.408 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:26.467 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:26.481+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:26.538 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:26.538 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:26.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:26.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T09:23:26.568 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:26.582+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:26.591 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:26.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:26.670 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:26.684+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:26.714 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:26.714 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:26.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:26.770 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:26.784+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:26.853 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:26.853 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:26.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:26.875 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:26.887+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:26.974 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:26.988+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:27.031 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:27.032 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:27.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:27.076 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:27.089+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:27.176 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:27.190+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:27.214 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:27.215 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:27.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:27.309 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:27.323+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:27.378 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:27.392+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:27.393 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:27.394 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:27.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:27.482 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:27.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:27.580 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:27.595+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:27.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:27.682 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:27.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:27.715 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:27.715 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:27.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:27.783 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:27.797+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:27.854 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:27.854 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:27.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:27.906 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:27.920+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:27.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:27.988 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:28.001+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:28.034 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:28.034 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:28.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:28.088 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:28.102+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:28.173 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:28.173 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:28.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:28.191 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:28.205+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:28.194 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:28.194 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:28.281 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:28.309 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:28.322+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:28.352 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:28.352 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:28.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:28.375 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:28.375 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:28.395 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:28.409+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:28.492 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:28.492 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:28.492 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:28.492 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:28.506 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:28.511+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:28.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:28.599 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:28.613+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:28.675 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:28.682 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:28.682 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:28.682 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:28.682 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:28.682 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:28.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:28.701 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:28.715+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:28.760 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:28.761 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:28.808 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:28.819+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:28.819 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:28.819 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:28.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:28.819 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:28.819 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:28.829 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:28.844 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:28.844 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:28.904 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:28.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:28.927 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:28.927 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:28.975 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:28.975 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:28.975 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:28.976 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:28.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:29.017 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:29.018 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:29.018 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:29.032+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:29.109 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:29.122+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:29.144 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:29.144 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:29.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:29.212 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:29.225+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:29.311 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:29.325+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:29.379 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:29.379 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:29.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:29.379 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:29.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.379 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 92.469912162, 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:29.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:29.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.386 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 90.052202045000001, 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 90.048017599999994, 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.392 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:29.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:29.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:29.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.962486514000005, 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:29.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:29.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:29.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:29.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:29.404 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.648509899000004, 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:29.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:29.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.646992054999998, 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.414 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:29.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:29.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.646326680000001, 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:29.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.642066507999999, 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:29.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.641433298999999, 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.437 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:29.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:29.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.640806197000003, 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:29.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:29.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.552546418999995, 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.451 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.456 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.551625439000006, 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:29.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:29.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:29.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:29.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 88.799247507999993, 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:29.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:29.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.929212536999998, 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.470 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:29.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:29.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:29.476 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.928039025999993, 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:29.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:29.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.482 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.919887055000004, 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:29.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:29.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.909605119000005, 2023-12-06T09:23:29.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:29.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:29.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:29.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.494 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.908947239, 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:29.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:29.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:29.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.908272460000006, 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.502 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:29.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:29.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 80.875914868999999, 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:29.514 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:29.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:29.516 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:29.530+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:29.618 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:29.632+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:29.682 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:29.682 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:29.682 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:29.683 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:29.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:29.719 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:29.733+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:29.825 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:29.835+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:29.859 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:29.860 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:29.860 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:29.860 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:29.860 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:29.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:29.922 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:29.936+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:30.005 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:30.005 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:30.005 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:30.005 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:30.006 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:30.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:30.023 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:30.037+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:30.125 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:30.138+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:30.186 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:30.186 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:30.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:30.225 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:30.239+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:23:30.297 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T09:23:30.297 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T09:23:30.297 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:23:30.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:30.326 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:30.340+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:30.326 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:30.326 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:30.327 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:30.327 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:30.327 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:30.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:30.427 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:30.441+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:30.514 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:30.514 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:30.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:30.514 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:30.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.553021099000006, 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:30.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:30.518 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:30.533 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:30.534 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:30.541+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:30.629 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:30.643+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:30.673 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:30.673 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:30.673 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:30.673 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:30.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:30.767 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:30.781+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:30.831 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:30.845+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:30.851 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:30.851 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:30.852 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:30.852 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:30.852 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:30.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:30.927 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:30.932 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:30.945+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:31.030 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:31.030 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:31.030 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:31.032 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:31.046+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:31.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:23:31.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:31.134 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:31.148+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:31.210 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:31.211 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:31.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:31.262 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:31.276+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:31.336 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:31.350+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:31.387 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:31.387 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:31.387 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:31.387 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:31.387 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:31.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:31.437 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:31.452+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:31.540 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:31.554+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:31.564 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 92.859898473000001, 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:31.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:31.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.568 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:31.569 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:31.569 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 92.165444604000001, 2023-12-06T09:23:31.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:31.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:31.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.573 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 92.063616023999998, 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.574 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:31.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.577 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.722552128000004, 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.578 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.579 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:31.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.583 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:31.585 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.713725847000006, 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:31.591 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.713104174999998, 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:31.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:31.598 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.705157693999993, 2023-12-06T09:23:31.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:31.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:31.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:31.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:31.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:31.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.614 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.704683676000002, 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.616 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.617 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:31.618 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.704229321, 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.619 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.620 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:31.621 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.622 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:31.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.320237070000005, 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:31.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 90.935087664999998, 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:31.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:31.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:31.633 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 90.922812354000001, 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:31.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:31.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:31.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:31.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 90.921326776000001, 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:31.642 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:31.643 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.014789617999995, 2023-12-06T09:23:31.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:31.643 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.643 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.643 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:31.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:31.648 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.009657314999998, 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.008831284999999, 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:31.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:31.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.006261824000006, 2023-12-06T09:23:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:31.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:31.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:31.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:31.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:31.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:31.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.668 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.005516353999994, 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.004710282999994, 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 82.993605830999996, 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:31.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.685 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.686 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:31.686 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:31.686 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:31.686 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:31.686 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:31.688 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:31.688 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:31.702+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:31.788 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:31.803+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:31.817 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:31.817 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:31.817 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:31.817 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:31.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:31.890 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:31.904+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:31.954 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:31.954 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:31.955 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:31.955 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:31.955 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:31.967 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:31.992 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:32.005+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:32.095 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:32.107+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:32.141 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:32.141 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:32.141 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:32.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:32.200 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:32.212+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:32.279 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:32.280 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:32.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:32.280 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:32.291 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:32.294 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:32.309+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:32.395 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:32.409+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:32.459 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:32.460 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:32.460 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:32.460 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:32.460 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:32.469 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:32.498 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:32.512+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:32.599 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 77.008545325, 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:32.600 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:32.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:32.603 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:32.604 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:32.605 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:32.614+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:32.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:32.700 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:32.715+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:32.748 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:32.748 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:32.748 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:32.748 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:32.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:32.802 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:32.816+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:32.884 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:32.884 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:32.884 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:32.884 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:32.884 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:32.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:32.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T09:23:32.903 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:32.917+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:33.004 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:33.018+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:33.021 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:33.021 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:33.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:33.021 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:33.021 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:33.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:33.105 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:33.119+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:33.198 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:33.198 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:33.198 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:33.199 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:33.208 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:33.208 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:33.222+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:33.309 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:33.323+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:33.336 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:33.336 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:33.336 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:33.336 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:33.336 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:33.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:33.410 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:33.424+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:33.511 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:33.524+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:33.527 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:33.527 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:33.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 96.623002146000005, 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:33.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:33.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 94.205292029000006, 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 94.201107583999999, 2023-12-06T09:23:33.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:33.540 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.540 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.540 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.540 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:33.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:33.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:33.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:33.546 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 94.115576497999996, 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.547 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.548 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.548 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.548 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:33.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.548 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:33.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:33.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:33.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.555 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.801599882999994, 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.556 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:33.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:33.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:33.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:33.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.800082039000003, 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.563 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:33.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.799416664000006, 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.570 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.571 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:33.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.571 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:33.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:33.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:33.577 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.795156492000004, 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.578 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:33.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:33.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.794523283000004, 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.585 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.793896180999994, 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.705636403, 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:33.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:33.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:33.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:33.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:33.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.704715422999996, 2023-12-06T09:23:33.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:33.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:33.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:33.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:33.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:33.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:33.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:33.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:33.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 92.952337491999998, 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:33.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.082302521000003, 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:33.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:33.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.631 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.081129009999998, 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:33.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:33.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.072977038999994, 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.639 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.062695102999996, 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.646 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:33.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:33.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.652 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.062037223000004, 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:33.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:33.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.061362443999997, 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.662 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:33.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:33.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:33.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:33.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 85.029004853000004, 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:33.668 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:33.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:33.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:33.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.673 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.674 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:33.674 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:33.674 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:33.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:33.677 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:33.690+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:33.777 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:33.791+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:33.805 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:33.805 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:33.805 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:33.805 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:33.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:33.900 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:33.914+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:33.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:33.986 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:33.986 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:33.986 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:33.986 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:33.986 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:33.986 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:34.000+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:33.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:34.081 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:34.094+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:34.170 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:34.170 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:34.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:34.170 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:34.170 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:34.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:34.182 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:34.195+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:34.282 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:34.296+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:34.318 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:34.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:34.318 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:34.318 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:34.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:34.413 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:34.427+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:34.489 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:34.503+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:34.506 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:34.506 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:34.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:34.608 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:34.619+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:34.685 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:34.700+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:34.687 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:34.687 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 79.726126923999999, 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:34.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:34.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:34.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:34.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:34.692 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:34.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:34.786 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:34.800+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:34.837 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:34.837 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:34.837 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:34.838 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:34.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:34.890 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:34.904+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:34.974 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:34.974 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:34.974 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:34.974 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:34.974 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:34.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:35.031 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:35.042+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:35.091 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:35.106+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:35.111 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:35.112 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:35.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:35.112 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:35.112 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:35.120 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:35.193 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:35.208+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:35.292 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:35.292 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:35.292 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:35.292 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:35.293 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:35.307+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:35.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:35.395 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:35.409+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:35.468 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:35.468 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:35.469 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:35.469 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:35.469 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:35.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:35.524 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:35.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:35.601 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:35.615+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:35.645 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:35.646 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 96.941254776999997, 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:35.650 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 96.246800907999997, 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 96.144972327999994, 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:35.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:35.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:35.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:35.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.660 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.660 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.803908432, 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:35.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:35.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.795082151000003, 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:35.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:35.679 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.794460478999994, 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:35.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:35.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.786513998000004, 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:35.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.786039979999998, 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:35.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.785585624999996, 2023-12-06T09:23:35.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.401593374000001, 2023-12-06T09:23:35.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:35.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.016443968999994, 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:35.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.722 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.004168657999998, 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.002683079999997, 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:35.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:35.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:35.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.096145922000005, 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:35.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:35.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:35.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:35.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.745 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.091013618999995, 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.090187588999996, 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:35.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.752 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.754 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.087618128000003, 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:35.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:35.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.086872658000004, 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.759 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:35.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.086066587000005, 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:35.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 87.074962135000007, 2023-12-06T09:23:35.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:35.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:35.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:35.771 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:35.773 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:35.773 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:35.787+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:35.873 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:35.888+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:35.943 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:35.943 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:35.943 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:35.943 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:35.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:35.975 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:35.989+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:36.076 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:36.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:36.124 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:36.124 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:36.124 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:36.124 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:36.124 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:36.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:36.177 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:36.190+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:36.277 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:36.291+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:36.301 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:36.301 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:36.301 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:36.301 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:36.301 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:36.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:36.378 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:36.392+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:36.439 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:36.439 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:36.439 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:36.439 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:36.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:36.488 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:36.502+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:36.581 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:36.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:36.616 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:36.616 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:36.616 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:36.616 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:36.616 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:36.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:36.681 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:36.695+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:36.691 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:36.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:36.691 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T09:23:36.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:36.754 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 81.162097031000002, 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:36.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:36.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:36.758 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:36.759 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:36.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:36.805 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:36.819+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:36.883 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:36.897+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:36.901 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:36.901 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:36.901 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:36.901 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:36.909 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:36.994 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:37.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:37.077 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:37.078 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:37.078 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:37.078 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:37.078 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:37.086 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:37.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:37.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:37.187 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:37.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:37.257 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:37.258 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:37.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:37.258 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:37.258 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:37.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:37.305 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:37.319+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:37.389 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:37.403+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:37.433 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:37.433 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:37.433 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:37.433 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:37.444 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:37.489 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:37.504+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:37.573 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:37.573 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:37.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:37.632 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:37.641+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:37.697 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:37.711+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:37.769 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:37.769 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:37.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:37.769 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 100.86020228700001, 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:37.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:37.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:37.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.777 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.778 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.442492169999994, 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.785 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.785 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:37.790 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.438307725000001, 2023-12-06T09:23:37.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:37.790 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.790 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.790 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:37.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:37.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.352776638999998, 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.796 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:37.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:37.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.038800023999997, 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:37.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:37.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.812 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.037282180000005, 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.817 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:37.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:37.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:37.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:37.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.823 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.036616804999994, 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:37.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:37.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.032356633000006, 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.829 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.836 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.837 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.031723424000006, 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:37.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:37.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:37.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.031096321999996, 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:37.843 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:37.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:37.848 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.942836544000002, 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.849 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:37.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:37.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.855 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.941915563999999, 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:37.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:37.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:37.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:37.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.860 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.189537633, 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:37.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.319502662000005, 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:37.867 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.318329151, 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:37.873 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:37.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:37.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.880 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.310177179999997, 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.883 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:37.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:37.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:37.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:37.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:37.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.889 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.890 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.299895243999998, 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:37.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:37.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.299237364000007, 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.896 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:37.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:37.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:37.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.298562584999999, 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:37.907 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.910 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.912 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.266204994000006, 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:37.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:37.917 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:37.918 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:37.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:37.920 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:37.934+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:38.020 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:38.035+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:38.087 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:38.087 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:38.087 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:38.087 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:38.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:38.122 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:38.135+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:38.222 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:38.236+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:38.224 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:38.224 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:38.224 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:38.224 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:38.225 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:38.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:38.323 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:38.338+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:38.406 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:38.406 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:38.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:38.406 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:38.406 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:38.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:38.460 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:38.474+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:38.529 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:38.543+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:38.554 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:38.554 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:38.554 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:38.554 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:38.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:38.649 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:38.663+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:38.732 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:38.733 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:38.733 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:38.746+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:38.740 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:38.833 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:38.847+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:38.909 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.948519078999993, 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:38.910 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:38.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:38.914 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:38.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:38.936 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:38.949+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:39.036 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:39.050+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:39.064 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:39.065 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:39.065 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:39.065 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:39.074 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:39.137 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:39.151+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:39.238 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:39.252+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:39.244 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:39.245 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:39.245 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:39.245 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:39.245 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:39.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:39.339 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:39.353+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:39.390 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:39.390 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:39.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:39.391 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:39.391 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:39.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:39.439 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:39.453+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:39.468 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":20,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:39.481 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:23:39.481 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T09:23:39.481 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T09:23:39.483 INFO:tasks.ceph.mon.b:Started 2023-12-06T09:23:39.483 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:23:39.483 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:39.532 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:39.533 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:39.533 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:39.533 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:39.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:39.570 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:39.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:39.645 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:39.659+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:39.719 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:39.719 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:39.719 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:39.719 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:39.719 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:39.731 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:39.775 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:39.789+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:39.851 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:39.865+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:39.907 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:39.907 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:39.908 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 101.198365634, 2023-12-06T09:23:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:39.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:39.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.914 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:39.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:39.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:39.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.503911765, 2023-12-06T09:23:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.920 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.921 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:39.921 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.921 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.921 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.921 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.926 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:39.926 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.926 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.930 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:39.931 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:39.931 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.402083185, 2023-12-06T09:23:39.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:39.932 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.932 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.932 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.932 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:39.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.935 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:39.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:39.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:39.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.939 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.939 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:39.939 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:39.940 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.940 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.941 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.941 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:39.941 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:39.941 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.061019289, 2023-12-06T09:23:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:39.943 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:39.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.052193008, 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.964 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.968 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.05157133599999, 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:39.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:39.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.043624855, 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:39.973 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.977 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.043150837, 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:39.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:39.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:39.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.042696482, 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.984 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.985 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.986 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.658704231000002, 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.987 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.990 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.273554825999994, 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:39.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:39.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.261279514999998, 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:39.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:39.999 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 99.259793936999998, 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:40.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:40.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:40.003 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.353256779000006, 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:40.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:40.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:40.007 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.348124475999995, 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:40.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:40.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:40.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:40.011 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.347298445999996, 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:40.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:40.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:40.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:40.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.344728985000003, 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.343983515000005, 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.343177444000005, 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:40.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.032 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 91.332072991999993, 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:40.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:40.037 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:40.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:40.039 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:40.053+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:40.140 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:40.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:40.211 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:40.211 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:40.212 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:40.212 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:40.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:40.242 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:40.256+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:40.297 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:40.343 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:40.358+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:40.365 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":20,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:40.403 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:40.403 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:40.405 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:40.405 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:40.406 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:40.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:40.443 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:40.457+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:40.512 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:40.512 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:40.544 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:40.559+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:40.586 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:40.587 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:40.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:40.587 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:40.587 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:40.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:40.599 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:40.599 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:40.627 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:23:40.705 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:40.705 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:40.706 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:40.720+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:40.745 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:40.760+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 39, 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:21:08.071156+0000", 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:23:40.064068+0000", 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "last_up_change": "2023-12-06T09:21:55.261571+0000", 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "last_in_change": "2023-12-06T09:23:15.949688+0000", 2023-12-06T09:23:40.780 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_num": 5799936, 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_set": [ 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "pglog_hardlimit", 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "purged_snapdirs", 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_deletes", 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "sortbitwise" 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_version": 6, 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T09:23:40.781 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_snapshot": "", 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_max": 2, 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "max_osd": 8, 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "require_min_compat_client": "luminous", 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "min_compat_client": "luminous", 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "require_osd_release": "reef", 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "allow_crimson": false, 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "pools": [ 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 1, 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": ".mgr", 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:21:54.386974+0000", 2023-12-06T09:23:40.782 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:23:40.783 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 1, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 1, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 1, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 1, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 1, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:23:40.784 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "18", 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:23:40.785 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:23:40.786 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:23:40.787 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:23:40.788 INFO:teuthology.orchestra.run.smithi088.stdout: "options": { 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_max": 32, 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_min": 1 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": {} 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 8, 2023-12-06T09:23:40.789 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 8, 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.25, 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 2, 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 2, 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 0.89483833312988281, 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 0.88999998569488525, 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2023-12-06T09:23:40.790 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 2, 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": "rbd", 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:22:01.832571+0000", 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:23:40.792 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:23:40.793 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:23:40.793 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:23:40.793 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:23:40.795 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:23:40.795 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:23:40.795 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:23:40.800 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:23:40.800 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:23:40.800 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 8, 2023-12-06T09:23:40.800 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 8, 2023-12-06T09:23:40.800 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 8, 2023-12-06T09:23:40.801 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 8, 2023-12-06T09:23:40.801 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 8, 2023-12-06T09:23:40.801 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:23:40.801 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:23:40.801 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "21", 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:23:40.803 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:23:40.804 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:23:40.804 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:23:40.804 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:23:40.805 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:23:40.806 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:23:40.807 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "options": {}, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd": {} 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 2.9800000190734863, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 2.9800000190734863, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.62999999523162842, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 1.8799999952316284, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 1.8799999952316284, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 0.89483833312988281, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 0.88999998569488525, 2023-12-06T09:23:40.808 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "db81b709-a318-4b43-beb2-bf3687f365e1", 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6805", 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:40.810 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6806", 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6808", 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.812 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6807", 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6805/2388339328", 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6806/2388339328", 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6808/2388339328", 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6807/2388339328", 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd7b0482-e237-424b-8eff-8db905713049", 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:40.814 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 15, 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6801", 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.815 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6802", 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6804", 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:40.816 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6803", 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6801/3646708602", 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6802/3646708602", 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6804/3646708602", 2023-12-06T09:23:40.817 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6803/3646708602", 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e5a571be-6b8e-4c6f-bd3e-2af30c112285", 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 35, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6810", 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:40.819 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6813", 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.820 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6816", 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6814", 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.821 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6810/3714168299", 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6813/3714168299", 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6816/3714168299", 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6814/3714168299", 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.822 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "d7abe19a-cc3d-411f-8533-8592e1379735", 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 0.1587066650390625, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 35, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.823 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6809", 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6811", 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.824 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6815", 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.825 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6812", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6809/1828251745", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6811/1828251745", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1828251745", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6812/1828251745", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.826 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b9517160-d081-44f9-aea4-6d2529e02114", 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 13, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 35, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.827 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6800", 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6801", 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.829 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6803", 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6802", 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.830 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6800/3720978101", 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6801/3720978101", 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6803/3720978101", 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6802/3720978101", 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:23:40.831 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e", 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 19, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.832 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6804", 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6805", 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6807", 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.835 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.835 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6806", 2023-12-06T09:23:40.835 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6804/1203998358", 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6805/1203998358", 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1203998358", 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6806/1203998358", 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e7afeca8-1363-4e9b-96f6-7fab6113d9e0", 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 30, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.837 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6808", 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.838 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6809", 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6811", 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.839 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6810", 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6808/4151455937", 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6809/4151455937", 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6811/4151455937", 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6810/4151455937", 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.840 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd4b3971-3adb-44fd-b42b-b6660305beb3", 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:23:40.841 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 26, 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.842 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6812", 2023-12-06T09:23:40.843 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:40.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.843 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.843 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:23:40.843 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6813", 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6815", 2023-12-06T09:23:40.844 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6814", 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4165054852 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.845 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.846 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6812/4165054852", 2023-12-06T09:23:40.846 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6813/4165054852", 2023-12-06T09:23:40.846 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6815/4165054852", 2023-12-06T09:23:40.846 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6814/4165054852", 2023-12-06T09:23:40.846 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_xinfo": [ 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.847 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 65536, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.229602+0000", 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.158659+0000", 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.969728+0000", 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.849 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.234010+0000", 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.850 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:50.601730+0000", 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.304649+0000", 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:23:40.851 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.671399+0000", 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:23:40.852 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.055036+0000", 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap": [ 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "pgid": "2.7", 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: 7, 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: 0 2023-12-06T09:23:40.853 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_items": [ 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "pgid": "2.1", 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "mappings": [ 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "from": 7, 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "to": 2 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_primaries": [], 2023-12-06T09:23:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_temp": [], 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_temp": [], 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklist": {}, 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "range_blocklist": {}, 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profiles": { 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "default": { 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "crush-failure-domain": "osd", 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "k": "2", 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "m": "1", 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin": "jerasure", 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "technique": "reed_sol_van" 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps_queue": [], 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "new_removed_snaps": [], 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "new_purged_snaps": [], 2023-12-06T09:23:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_node_flags": {}, 2023-12-06T09:23:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: "device_class_flags": {}, 2023-12-06T09:23:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": { 2023-12-06T09:23:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_enabled": false, 2023-12-06T09:23:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_bucket_count": 0, 2023-12-06T09:23:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: "degraded_stretch_mode": 0, 2023-12-06T09:23:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: "recovering_stretch_mode": 0, 2023-12-06T09:23:40.857 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_bucket": 0 2023-12-06T09:23:40.857 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:40.857 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:40.859 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:40.859 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:40.860 INFO:tasks.thrashosds.thrasher:j is {'epoch': 39, 'fsid': '0c48cca3-dcc0-44bd-8eed-37fea2138550', 'created': '2023-12-06T09:21:08.071156+0000', 'modified': '2023-12-06T09:23:40.064068+0000', 'last_up_change': '2023-12-06T09:21:55.261571+0000', 'last_in_change': '2023-12-06T09:23:15.949688+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T09:21:54.386974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8948383331298828, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T09:22:01.832571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.9800000190734863, 'score_stable': 2.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8948383331298828, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'db81b709-a318-4b43-beb2-bf3687f365e1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6805', 'nonce': 2388339328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6806', 'nonce': 2388339328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6808', 'nonce': 2388339328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6807', 'nonce': 2388339328}]}, 'public_addr': '172.21.15.88:6805/2388339328', 'cluster_addr': '172.21.15.88:6806/2388339328', 'heartbeat_back_addr': '172.21.15.88:6808/2388339328', 'heartbeat_front_addr': '172.21.15.88:6807/2388339328', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'fd7b0482-e237-424b-8eff-8db905713049', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6801', 'nonce': 3646708602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6802', 'nonce': 3646708602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6804', 'nonce': 3646708602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6803', 'nonce': 3646708602}]}, 'public_addr': '172.21.15.88:6801/3646708602', 'cluster_addr': '172.21.15.88:6802/3646708602', 'heartbeat_back_addr': '172.21.15.88:6804/3646708602', 'heartbeat_front_addr': '172.21.15.88:6803/3646708602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e5a571be-6b8e-4c6f-bd3e-2af30c112285', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6810', 'nonce': 3714168299}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6813', 'nonce': 3714168299}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6816', 'nonce': 3714168299}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6814', 'nonce': 3714168299}]}, 'public_addr': '172.21.15.88:6810/3714168299', 'cluster_addr': '172.21.15.88:6813/3714168299', 'heartbeat_back_addr': '172.21.15.88:6816/3714168299', 'heartbeat_front_addr': '172.21.15.88:6814/3714168299', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd7abe19a-cc3d-411f-8533-8592e1379735', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1587066650390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6809', 'nonce': 1828251745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6811', 'nonce': 1828251745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6815', 'nonce': 1828251745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6812', 'nonce': 1828251745}]}, 'public_addr': '172.21.15.88:6809/1828251745', 'cluster_addr': '172.21.15.88:6811/1828251745', 'heartbeat_back_addr': '172.21.15.88:6815/1828251745', 'heartbeat_front_addr': '172.21.15.88:6812/1828251745', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b9517160-d081-44f9-aea4-6d2529e02114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 3720978101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6801', 'nonce': 3720978101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6803', 'nonce': 3720978101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 3720978101}]}, 'public_addr': '172.21.15.132:6800/3720978101', 'cluster_addr': '172.21.15.132:6801/3720978101', 'heartbeat_back_addr': '172.21.15.132:6803/3720978101', 'heartbeat_front_addr': '172.21.15.132:6802/3720978101', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1203998358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6805', 'nonce': 1203998358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6807', 'nonce': 1203998358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1203998358}]}, 'public_addr': '172.21.15.132:6804/1203998358', 'cluster_addr': '172.21.15.132:6805/1203998358', 'heartbeat_back_addr': '172.21.15.132:6807/1203998358', 'heartbeat_front_addr': '172.21.15.132:6806/1203998358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7afeca8-1363-4e9b-96f6-7fab6113d9e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 4151455937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6809', 'nonce': 4151455937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6811', 'nonce': 4151455937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 4151455937}]}, 'public_addr': '172.21.15.132:6808/4151455937', 'cluster_addr': '172.21.15.132:6809/4151455937', 'heartbeat_back_addr': '172.21.15.132:6811/4151455937', 'heartbeat_front_addr': '172.21.15.132:6810/4151455937', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fd4b3971-3adb-44fd-b42b-b6660305beb3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 4165054852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6813', 'nonce': 4165054852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6815', 'nonce': 4165054852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 4165054852}]}, 'public_addr': '172.21.15.132:6812/4165054852', 'cluster_addr': '172.21.15.132:6813/4165054852', 'heartbeat_back_addr': '172.21.15.132:6815/4165054852', 'heartbeat_front_addr': '172.21.15.132:6814/4165054852', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.229602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.158659+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.969728+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.234010+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:50.601730+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.304649+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.671399+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.055036+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.7', 'osds': [7, 0]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T09:23:40.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:23:40.861 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:40.861 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:40.862 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:40.862 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:40.872+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:40.909 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:40.910 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:40.959 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:40.973+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:41.016 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:41.016 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:41.053 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:41.054 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:41.060 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:41.075+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:41.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:41.168 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:41.181+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:41.252 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:41.252 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:41.262 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:41.263 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:41.276+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:41.363 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:41.378+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:41.391 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:41.392 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:41.400 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:41.459 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:23:41.459 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:23:41.464 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:41.479+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:41.472 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2023-12-06T09:23:40.668709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":6,"kb":660602880,"kb_used":191648,"kb_used_data":2508,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660411232,"statfs":{"total":676457349120,"available":676261101568,"internally_reserved":0,"allocated":2568192,"data_stored":1742563,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":11157,"internal_metadata":193582187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.732099"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.645962+0000","last_change":"2023-12-06T09:23:17.838365+0000","last_active":"2023-12-06T09:23:18.645962+0000","last_peered":"2023-12-06T09:23:18.645962+0000","last_clean":"2023-12-06T09:23:18.645962+0000","last_became_active":"2023-12-06T09:23:17.831349+0000","last_became_peered":"2023-12-06T09:23:17.831349+0000","last_unstale":"2023-12-06T09:23:18.645962+0000","last_undegraded":"2023-12-06T09:23:18.645962+0000","last_fullsized":"2023-12-06T09:23:18.645962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:11:12.082222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.698572+0000","last_change":"2023-12-06T09:23:17.800196+0000","last_active":"2023-12-06T09:23:18.698572+0000","last_peered":"2023-12-06T09:23:18.698572+0000","last_clean":"2023-12-06T09:23:18.698572+0000","last_became_active":"2023-12-06T09:23:17.793541+0000","last_became_peered":"2023-12-06T09:23:17.793541+0000","last_unstale":"2023-12-06T09:23:18.698572+0000","last_undegraded":"2023-12-06T09:23:18.698572+0000","last_fullsized":"2023-12-06T09:23:18.698572+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:08:40.625364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.661108+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:23:18.661108+0000","last_peered":"2023-12-06T09:23:18.661108+0000","last_clean":"2023-12-06T09:23:18.661108+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:23:18.661108+0000","last_undegraded":"2023-12-06T09:23:18.661108+0000","last_fullsized":"2023-12-06T09:23:18.661108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.694844+0000","last_change":"2023-12-06T09:22:04.222023+0000","last_active":"2023-12-06T09:23:18.694844+0000","last_peered":"2023-12-06T09:23:18.694844+0000","last_clean":"2023-12-06T09:23:18.694844+0000","last_became_active":"2023-12-06T09:22:04.183213+0000","last_became_peered":"2023-12-06T09:22:04.183213+0000","last_unstale":"2023-12-06T09:23:18.694844+0000","last_undegraded":"2023-12-06T09:23:18.694844+0000","last_fullsized":"2023-12-06T09:23:18.694844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:22:13.132330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.693554+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:23:18.693554+0000","last_peered":"2023-12-06T09:23:18.693554+0000","last_clean":"2023-12-06T09:23:18.693554+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:23:18.693554+0000","last_undegraded":"2023-12-06T09:23:18.693554+0000","last_fullsized":"2023-12-06T09:23:18.693554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.541915+0000","last_change":"2023-12-06T09:23:17.798060+0000","last_active":"2023-12-06T09:23:18.541915+0000","last_peered":"2023-12-06T09:23:18.541915+0000","last_clean":"2023-12-06T09:23:18.541915+0000","last_became_active":"2023-12-06T09:23:17.791978+0000","last_became_peered":"2023-12-06T09:23:17.791978+0000","last_unstale":"2023-12-06T09:23:18.541915+0000","last_undegraded":"2023-12-06T09:23:18.541915+0000","last_fullsized":"2023-12-06T09:23:18.541915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:13.702364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.649712+0000","last_change":"2023-12-06T09:22:03.754841+0000","last_active":"2023-12-06T09:23:18.649712+0000","last_peered":"2023-12-06T09:23:18.649712+0000","last_clean":"2023-12-06T09:23:18.649712+0000","last_became_active":"2023-12-06T09:22:03.725823+0000","last_became_peered":"2023-12-06T09:22:03.725823+0000","last_unstale":"2023-12-06T09:23:18.649712+0000","last_undegraded":"2023-12-06T09:23:18.649712+0000","last_fullsized":"2023-12-06T09:23:18.649712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:11:51.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.544102+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:23:18.544102+0000","last_peered":"2023-12-06T09:23:18.544102+0000","last_clean":"2023-12-06T09:23:18.544102+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:23:18.544102+0000","last_undegraded":"2023-12-06T09:23:18.544102+0000","last_fullsized":"2023-12-06T09:23:18.544102+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":132,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-06T09:23:18.709056+0000","last_change":"2023-12-06T09:21:58.354594+0000","last_active":"2023-12-06T09:23:18.709056+0000","last_peered":"2023-12-06T09:23:18.709056+0000","last_clean":"2023-12-06T09:23:18.709056+0000","last_became_active":"2023-12-06T09:21:58.323313+0000","last_became_peered":"2023-12-06T09:21:58.323313+0000","last_unstale":"2023-12-06T09:23:18.709056+0000","last_undegraded":"2023-12-06T09:23:18.709056+0000","last_fullsized":"2023-12-06T09:23:18.709056+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:39:56.605171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":1378,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":17.015000000000001,"15min":17.015000000000001},"min":{"1min":5.3929999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"max":{"1min":49.691000000000003,"5min":49.691000000000003,"15min":49.691000000000003},"last":22.085999999999999},{"interface":"front","average":{"1min":17.239999999999998,"5min":17.239999999999998,"15min":17.239999999999998},"min":{"1min":6.266,"5min":6.266,"15min":6.266},"max":{"1min":51.530000000000001,"5min":51.530000000000001,"15min":51.530000000000001},"last":20.541}]},{"osd":1,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":12.565,"5min":12.565,"15min":12.565},"min":{"1min":5.9649999999999999,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":38.488999999999997,"5min":38.488999999999997,"15min":38.488999999999997},"last":7.5199999999999996},{"interface":"front","average":{"1min":15.016999999999999,"5min":15.016999999999999,"15min":15.016999999999999},"min":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"max":{"1min":35.378999999999998,"5min":35.378999999999998,"15min":35.378999999999998},"last":13.262}]},{"osd":2,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":15.605,"5min":15.605,"15min":15.605},"min":{"1min":7.2000000000000002,"5min":7.2000000000000002,"15min":7.2000000000000002},"max":{"1min":42.523000000000003,"5min":42.523000000000003,"15min":42.523000000000003},"last":8.9139999999999997},{"interface":"front","average":{"1min":18.297999999999998,"5min":18.297999999999998,"15min":18.297999999999998},"min":{"1min":9.5459999999999994,"5min":9.5459999999999994,"15min":9.5459999999999994},"max":{"1min":39.066000000000003,"5min":39.066000000000003,"15min":39.066000000000003},"last":12.334}]},{"osd":4,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":15.315,"5min":15.315,"15min":15.315},"min":{"1min":8.9199999999999999,"5min":8.9199999999999999,"15min":8.9199999999999999},"max":{"1min":25.123000000000001,"5min":25.123000000000001,"15min":25.123000000000001},"last":6.4720000000000004},{"interface":"front","average":{"1min":15.606,"5min":15.606,"15min":15.606},"min":{"1min":7.7880000000000003,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":25.466000000000001,"5min":25.466000000000001,"15min":25.466000000000001},"last":10.875999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":27.913,"5min":27.913,"15min":27.913},"min":{"1min":8.298,"5min":8.298,"15min":8.298},"max":{"1min":190.821,"5min":190.821,"15min":190.821},"last":8.359},{"interface":"front","average":{"1min":27.640999999999998,"5min":27.640999999999998,"15min":27.640999999999998},"min":{"1min":8.5969999999999995,"5min":8.5969999999999995,"15min":8.5969999999999995},"max":{"1min":191.86199999999999,"5min":191.86199999999999,"15min":191.86199999999999},"last":14.173}]},{"osd":6,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":27.119,"5min":27.119,"15min":27.119},"min":{"1min":12.289,"5min":12.289,"15min":12.289},"max":{"1min":166.45400000000001,"5min":166.45400000000001,"15min":166.45400000000001},"last":9.5700000000000003},{"interface":"front","average":{"1min":27.024999999999999,"5min":27.024999999999999,"15min":27.024999999999999},"min":{"1min":11.528,"5min":11.528,"15min":11.528},"max":{"1min":117.58499999999999,"5min":117.58499999999999,"15min":117.58499999999999},"last":15.087999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:22:58 2023","interfaces":[{"interface":"back","average":{"1min":18.994,"5min":18.994,"15min":18.994},"min":{"1min":6.6619999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":32.558,"5min":32.558,"15min":32.558},"last":16.029},{"interface":"front","average":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":19.864000000000001},"min":{"1min":12.651,"5min":12.651,"15min":12.651},"max":{"1min":27.971,"5min":27.971,"15min":27.971},"last":11.502000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509463,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":7.7359999999999998,"5min":7.7359999999999998,"15min":7.7359999999999998},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":18.952000000000002,"5min":18.952000000000002,"15min":18.952000000000002},"last":7.6950000000000003},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.356999999999999,"15min":12.356999999999999},"min":{"1min":7.0049999999999999,"5min":7.0049999999999999,"15min":7.0049999999999999},"max":{"1min":19.617999999999999,"5min":19.617999999999999,"15min":19.617999999999999},"last":11.087}]},{"osd":1,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":15.377000000000001,"5min":15.377000000000001,"15min":15.377000000000001},"min":{"1min":7.319,"5min":7.319,"15min":7.319},"max":{"1min":52.530000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":14.19},{"interface":"front","average":{"1min":18.968,"5min":18.968,"15min":18.968},"min":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":54.338000000000001,"5min":54.338000000000001,"15min":54.338000000000001},"last":15.955}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":10.951000000000001,"5min":10.951000000000001,"15min":10.951000000000001},"min":{"1min":6.5129999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"max":{"1min":27.295999999999999,"5min":27.295999999999999,"15min":27.295999999999999},"last":10.805},{"interface":"front","average":{"1min":13.555999999999999,"5min":13.555999999999999,"15min":13.555999999999999},"min":{"1min":6.0460000000000003,"5min":6.0460000000000003,"15min":6.0460000000000003},"max":{"1min":21.73,"5min":21.73,"15min":21.73},"last":11.406000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":15.212999999999999,"5min":15.212999999999999,"15min":15.212999999999999},"min":{"1min":7.7009999999999996,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":32.063000000000002,"5min":32.063000000000002,"15min":32.063000000000002},"last":13.233000000000001},{"interface":"front","average":{"1min":17.879000000000001,"5min":17.879000000000001,"15min":17.879000000000001},"min":{"1min":9.2919999999999998,"5min":9.2919999999999998,"15min":9.2919999999999998},"max":{"1min":36.337000000000003,"5min":36.337000000000003,"15min":36.337000000000003},"last":19.324000000000002}]},{"osd":5,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"min":{"1min":8.4900000000000002,"5min":8.4900000000000002,"15min":8.4900000000000002},"max":{"1min":56.491,"5min":56.491,"15min":56.491},"last":8.8989999999999991},{"interface":"front","average":{"1min":20.263000000000002,"5min":20.263000000000002,"15min":20.263000000000002},"min":{"1min":9.3719999999999999,"5min":9.3719999999999999,"15min":9.3719999999999999},"max":{"1min":75.540999999999997,"5min":75.540999999999997,"15min":75.540999999999997},"last":18.283000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":18.440999999999999,"5min":18.440999999999999,"15min":18.440999999999999},"min":{"1min":11.493,"5min":11.493,"15min":11.493},"max":{"1min":30.213999999999999,"5min":30.213999999999999,"15min":30.213999999999999},"last":17.274000000000001},{"interface":"front","average":{"1min":14.678000000000001,"5min":14.678000000000001,"15min":14.678000000000001},"min":{"1min":7.6100000000000003,"5min":7.6100000000000003,"15min":7.6100000000000003},"max":{"1min":25.847999999999999,"5min":25.847999999999999,"15min":25.847999999999999},"last":12.651}]},{"osd":7,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":20.239000000000001,"5min":20.239000000000001,"15min":20.239000000000001},"min":{"1min":13.896000000000001,"5min":13.896000000000001,"15min":13.896000000000001},"max":{"1min":35.094000000000001,"5min":35.094000000000001,"15min":35.094000000000001},"last":19.937999999999999},{"interface":"front","average":{"1min":18.385000000000002,"5min":18.385000000000002,"15min":18.385000000000002},"min":{"1min":10.811,"5min":10.811,"15min":10.811},"max":{"1min":27.387,"5min":27.387,"15min":27.387},"last":15.266}]}]},{"osd":0,"up_from":0,"seq":64424509464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":10.112,"5min":10.112,"15min":10.112},"min":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":25.277999999999999,"5min":25.277999999999999,"15min":25.277999999999999},"last":17.606000000000002},{"interface":"front","average":{"1min":10.464,"5min":10.464,"15min":10.464},"min":{"1min":5.117,"5min":5.117,"15min":5.117},"max":{"1min":28.972000000000001,"5min":28.972000000000001,"15min":28.972000000000001},"last":7.6529999999999996}]},{"osd":1,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":16.478999999999999,"5min":16.478999999999999,"15min":16.478999999999999},"min":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":5.6849999999999996},"max":{"1min":57.503,"5min":57.503,"15min":57.503},"last":15.922000000000001},{"interface":"front","average":{"1min":17.050999999999998,"5min":17.050999999999998,"15min":17.050999999999998},"min":{"1min":7.3070000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":47.421999999999997,"5min":47.421999999999997,"15min":47.421999999999997},"last":6.8639999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":16.943000000000001,"5min":16.943000000000001,"15min":16.943000000000001},"min":{"1min":6.9189999999999996,"5min":6.9189999999999996,"15min":6.9189999999999996},"max":{"1min":41.762999999999998,"5min":41.762999999999998,"15min":41.762999999999998},"last":8.298},{"interface":"front","average":{"1min":12.894,"5min":12.894,"15min":12.894},"min":{"1min":6.3070000000000004,"5min":6.3070000000000004,"15min":6.3070000000000004},"max":{"1min":42.520000000000003,"5min":42.520000000000003,"15min":42.520000000000003},"last":13.997999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":6.6520000000000001,"5min":6.6520000000000001,"15min":6.6520000000000001},"max":{"1min":33.357999999999997,"5min":33.357999999999997,"15min":33.357999999999997},"last":19.102},{"interface":"front","average":{"1min":16.074999999999999,"5min":16.074999999999999,"15min":16.074999999999999},"min":{"1min":5.7949999999999999,"5min":5.7949999999999999,"15min":5.7949999999999999},"max":{"1min":36.402999999999999,"5min":36.402999999999999,"15min":36.402999999999999},"last":15.057}]},{"osd":4,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":15.061,"5min":15.061,"15min":15.061},"min":{"1min":6.9160000000000004,"5min":6.9160000000000004,"15min":6.9160000000000004},"max":{"1min":32.531999999999996,"5min":32.531999999999996,"15min":32.531999999999996},"last":16.597000000000001},{"interface":"front","average":{"1min":16.821999999999999,"5min":16.821999999999999,"15min":16.821999999999999},"min":{"1min":8.0370000000000008,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":39.546999999999997,"5min":39.546999999999997,"15min":39.546999999999997},"last":9.6530000000000005}]},{"osd":5,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":21.998999999999999,"5min":21.998999999999999,"15min":21.998999999999999},"min":{"1min":9.218,"5min":9.218,"15min":9.218},"max":{"1min":77.706999999999994,"5min":77.706999999999994,"15min":77.706999999999994},"last":8.9299999999999997},{"interface":"front","average":{"1min":22.5,"5min":22.5,"15min":22.5},"min":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"max":{"1min":69.787999999999997,"5min":69.787999999999997,"15min":69.787999999999997},"last":12.345000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":20.309000000000001,"5min":20.309000000000001,"15min":20.309000000000001},"min":{"1min":10.097,"5min":10.097,"15min":10.097},"max":{"1min":62.383000000000003,"5min":62.383000000000003,"15min":62.383000000000003},"last":18.318000000000001},{"interface":"front","average":{"1min":21.812000000000001,"5min":21.812000000000001,"15min":21.812000000000001},"min":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"max":{"1min":63.496000000000002,"5min":63.496000000000002,"15min":63.496000000000002},"last":11.34}]}]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":12.169,"5min":12.169,"15min":12.169},"min":{"1min":4.218,"5min":4.218,"15min":4.218},"max":{"1min":45.506,"5min":45.506,"15min":45.506},"last":4.6280000000000001},{"interface":"front","average":{"1min":13.605,"5min":13.605,"15min":13.605},"min":{"1min":5.9530000000000003,"5min":5.9530000000000003,"15min":5.9530000000000003},"max":{"1min":46.450000000000003,"5min":46.450000000000003,"15min":46.450000000000003},"last":9.1890000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":16.016999999999999,"5min":16.016999999999999,"15min":16.016999999999999},"min":{"1min":7.2140000000000004,"5min":7.2140000000000004,"15min":7.2140000000000004},"max":{"1min":55.398000000000003,"5min":55.398000000000003,"15min":55.398000000000003},"last":10.68},{"interface":"front","average":{"1min":15.51,"5min":15.51,"15min":15.51},"min":{"1min":7.8559999999999999,"5min":7.8559999999999999,"15min":7.8559999999999999},"max":{"1min":41.811999999999998,"5min":41.811999999999998,"15min":41.811999999999998},"last":9.4979999999999993}]},{"osd":2,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":22.890000000000001,"5min":22.890000000000001,"15min":22.890000000000001},"min":{"1min":5.1420000000000003,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":208.93799999999999,"5min":208.93799999999999,"15min":208.93799999999999},"last":11.91},{"interface":"front","average":{"1min":27.946000000000002,"5min":27.946000000000002,"15min":27.946000000000002},"min":{"1min":10.807,"5min":10.807,"15min":10.807},"max":{"1min":208.57400000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":17.298999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":18.024000000000001,"5min":18.024000000000001,"15min":18.024000000000001},"min":{"1min":7.032,"5min":7.032,"15min":7.032},"max":{"1min":43.276000000000003,"5min":43.276000000000003,"15min":43.276000000000003},"last":15.462999999999999},{"interface":"front","average":{"1min":15.728999999999999,"5min":15.728999999999999,"15min":15.728999999999999},"min":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"max":{"1min":41.493000000000002,"5min":41.493000000000002,"15min":41.493000000000002},"last":8.2270000000000003}]},{"osd":4,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":25.762,"5min":25.762,"15min":25.762},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":117.10299999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":16.445},{"interface":"front","average":{"1min":28.823,"5min":28.823,"15min":28.823},"min":{"1min":9.1460000000000008,"5min":9.1460000000000008,"15min":9.1460000000000008},"max":{"1min":111.874,"5min":111.874,"15min":111.874},"last":14.696}]},{"osd":5,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":18.038,"5min":18.038,"15min":18.038},"min":{"1min":8.0410000000000004,"5min":8.0410000000000004,"15min":8.0410000000000004},"max":{"1min":39.886000000000003,"5min":39.886000000000003,"15min":39.886000000000003},"last":11.257},{"interface":"front","average":{"1min":23.875,"5min":23.875,"15min":23.875},"min":{"1min":13.242000000000001,"5min":13.242000000000001,"15min":13.242000000000001},"max":{"1min":60.530999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":18.082999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":25.402999999999999,"5min":25.402999999999999,"15min":25.402999999999999},"min":{"1min":9.6839999999999993,"5min":9.6839999999999993,"15min":9.6839999999999993},"max":{"1min":87.590999999999994,"5min":87.590999999999994,"15min":87.590999999999994},"last":13.255000000000001},{"interface":"front","average":{"1min":21.391999999999999,"5min":21.391999999999999,"15min":21.391999999999999},"min":{"1min":9.6929999999999996,"5min":9.6929999999999996,"15min":9.6929999999999996},"max":{"1min":50.737000000000002,"5min":50.737000000000002,"15min":50.737000000000002},"last":12.599}]}]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":533109,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.371,"15min":14.371},"min":{"1min":5.4210000000000003,"5min":5.4210000000000003,"15min":5.4210000000000003},"max":{"1min":45.491,"5min":45.491,"15min":45.491},"last":8.6989999999999998},{"interface":"front","average":{"1min":13.964,"5min":13.964,"15min":13.964},"min":{"1min":4.2859999999999996,"5min":4.2859999999999996,"15min":4.2859999999999996},"max":{"1min":49.991999999999997,"5min":49.991999999999997,"15min":49.991999999999997},"last":6.1639999999999997}]},{"osd":1,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":25.047000000000001,"5min":25.047000000000001,"15min":25.047000000000001},"min":{"1min":4.6710000000000003,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":220.13900000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":9.3919999999999995},{"interface":"front","average":{"1min":35.741999999999997,"5min":35.741999999999997,"15min":35.741999999999997},"min":{"1min":7.476,"5min":7.476,"15min":7.476},"max":{"1min":344.79599999999999,"5min":344.79599999999999,"15min":344.79599999999999},"last":12.59}]},{"osd":2,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":13.627000000000001,"5min":13.627000000000001,"15min":13.627000000000001},"min":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":53.235999999999997,"5min":53.235999999999997,"15min":53.235999999999997},"last":15.601000000000001},{"interface":"front","average":{"1min":15.888999999999999,"5min":15.888999999999999,"15min":15.888999999999999},"min":{"1min":9.1989999999999998,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":54.747,"5min":54.747,"15min":54.747},"last":13.226000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":13.504,"5min":13.504,"15min":13.504},"min":{"1min":6.4189999999999996,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":46.685000000000002,"5min":46.685000000000002,"15min":46.685000000000002},"last":16.756},{"interface":"front","average":{"1min":17.748999999999999,"5min":17.748999999999999,"15min":17.748999999999999},"min":{"1min":8.8670000000000009,"5min":8.8670000000000009,"15min":8.8670000000000009},"max":{"1min":51.045999999999999,"5min":51.045999999999999,"15min":51.045999999999999},"last":8.3930000000000007}]},{"osd":4,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":22.210999999999999,"5min":22.210999999999999,"15min":22.210999999999999},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":281.608,"5min":281.608,"15min":281.608},"last":10.529999999999999},{"interface":"front","average":{"1min":24.216000000000001,"5min":24.216000000000001,"15min":24.216000000000001},"min":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":7.7199999999999998},"max":{"1min":225.83600000000001,"5min":225.83600000000001,"15min":225.83600000000001},"last":14.269}]},{"osd":6,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":22.564,"5min":22.564,"15min":22.564},"min":{"1min":8.4109999999999996,"5min":8.4109999999999996,"15min":8.4109999999999996},"max":{"1min":198.667,"5min":198.667,"15min":198.667},"last":9.9459999999999997},{"interface":"front","average":{"1min":24.366,"5min":24.366,"15min":24.366},"min":{"1min":9.1699999999999999,"5min":9.1699999999999999,"15min":9.1699999999999999},"max":{"1min":208.446,"5min":208.446,"15min":208.446},"last":17.533999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:22:59 2023","interfaces":[{"interface":"back","average":{"1min":27.131,"5min":27.131,"15min":27.131},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":229.83500000000001,"5min":229.83500000000001,"15min":229.83500000000001},"last":14.901999999999999},{"interface":"front","average":{"1min":24.824999999999999,"5min":24.824999999999999,"15min":24.824999999999999},"min":{"1min":8.5340000000000007,"5min":8.5340000000000007,"15min":8.5340000000000007},"max":{"1min":233.59700000000001,"5min":233.59700000000001,"15min":233.59700000000001},"last":11.125}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":533109,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":11.372999999999999,"5min":11.372999999999999,"15min":11.372999999999999},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":40.225999999999999,"5min":40.225999999999999,"15min":40.225999999999999},"last":8.6219999999999999},{"interface":"front","average":{"1min":12.084,"5min":12.084,"15min":12.084},"min":{"1min":4.4950000000000001,"5min":4.4950000000000001,"15min":4.4950000000000001},"max":{"1min":30.405000000000001,"5min":30.405000000000001,"15min":30.405000000000001},"last":6.3179999999999996}]},{"osd":2,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":10.69,"5min":10.69,"15min":10.69},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":23.539999999999999,"5min":23.539999999999999,"15min":23.539999999999999},"last":12.470000000000001},{"interface":"front","average":{"1min":15.456,"5min":15.456,"15min":15.456},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":45.158000000000001,"5min":45.158000000000001,"15min":45.158000000000001},"last":10.595000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:22:56 2023","interfaces":[{"interface":"back","average":{"1min":22.263999999999999,"5min":22.263999999999999,"15min":22.263999999999999},"min":{"1min":5.8230000000000004,"5min":5.8230000000000004,"15min":5.8230000000000004},"max":{"1min":122.762,"5min":122.762,"15min":122.762},"last":7.2279999999999998},{"interface":"front","average":{"1min":21.706,"5min":21.706,"15min":21.706},"min":{"1min":7.742,"5min":7.742,"15min":7.742},"max":{"1min":129.99100000000001,"5min":129.99100000000001,"15min":129.99100000000001},"last":7.8970000000000002}]},{"osd":4,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":32.768999999999998,"5min":32.768999999999998,"15min":32.768999999999998},"min":{"1min":7.944,"5min":7.944,"15min":7.944},"max":{"1min":359.46300000000002,"5min":359.46300000000002,"15min":359.46300000000002},"last":14.721},{"interface":"front","average":{"1min":31.254999999999999,"5min":31.254999999999999,"15min":31.254999999999999},"min":{"1min":8.1229999999999993,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":314.47699999999998,"5min":314.47699999999998,"15min":314.47699999999998},"last":16.263999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":28.239000000000001,"5min":28.239000000000001,"15min":28.239000000000001},"min":{"1min":9.9770000000000003,"5min":9.9770000000000003,"15min":9.9770000000000003},"max":{"1min":152.97399999999999,"5min":152.97399999999999,"15min":152.97399999999999},"last":10.053000000000001},{"interface":"front","average":{"1min":26.779,"5min":26.779,"15min":26.779},"min":{"1min":10.106,"5min":10.106,"15min":10.106},"max":{"1min":154.53100000000001,"5min":154.53100000000001,"15min":154.53100000000001},"last":15.515000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":17.939,"5min":17.939,"15min":17.939},"min":{"1min":8.3399999999999999,"5min":8.3399999999999999,"15min":8.3399999999999999},"max":{"1min":43.045000000000002,"5min":43.045000000000002,"15min":43.045000000000002},"last":11.757999999999999},{"interface":"front","average":{"1min":21.443000000000001,"5min":21.443000000000001,"15min":21.443000000000001},"min":{"1min":11.129,"5min":11.129,"15min":11.129},"max":{"1min":41.356000000000002,"5min":41.356000000000002,"15min":41.356000000000002},"last":13.946}]},{"osd":7,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":34.292000000000002,"5min":34.292000000000002,"15min":34.292000000000002},"min":{"1min":13.757,"5min":13.757,"15min":13.757},"max":{"1min":285.81700000000001,"5min":285.81700000000001,"15min":285.81700000000001},"last":17.108000000000001},{"interface":"front","average":{"1min":33.560000000000002,"5min":33.560000000000002,"15min":33.560000000000002},"min":{"1min":9.3260000000000005,"5min":9.3260000000000005,"15min":9.3260000000000005},"max":{"1min":295.28500000000003,"5min":295.28500000000003,"15min":295.28500000000003},"last":11.202999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":14.59,"5min":14.59,"15min":14.59},"min":{"1min":4.3079999999999998,"5min":4.3079999999999998,"15min":4.3079999999999998},"max":{"1min":120.748,"5min":120.748,"15min":120.748},"last":9.3680000000000003},{"interface":"front","average":{"1min":20.132000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"min":{"1min":4.2930000000000001,"5min":4.2930000000000001,"15min":4.2930000000000001},"max":{"1min":199.268,"5min":199.268,"15min":199.268},"last":7.0570000000000004}]},{"osd":1,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":20.681999999999999,"15min":20.681999999999999},"min":{"1min":7.3860000000000001,"5min":7.3860000000000001,"15min":7.3860000000000001},"max":{"1min":92.817999999999998,"5min":92.817999999999998,"15min":92.817999999999998},"last":12.555999999999999},{"interface":"front","average":{"1min":21.934000000000001,"5min":21.934000000000001,"15min":21.934000000000001},"min":{"1min":7.1989999999999998,"5min":7.1989999999999998,"15min":7.1989999999999998},"max":{"1min":95.665999999999997,"5min":95.665999999999997,"15min":95.665999999999997},"last":7.7489999999999997}]},{"osd":2,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":25.824999999999999,"5min":25.824999999999999,"15min":25.824999999999999},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":208.06,"5min":208.06,"15min":208.06},"last":13.276},{"interface":"front","average":{"1min":19.771999999999998,"5min":19.771999999999998,"15min":19.771999999999998},"min":{"1min":6.915,"5min":6.915,"15min":6.915},"max":{"1min":125.047,"5min":125.047,"15min":125.047},"last":8.4819999999999993}]},{"osd":3,"last update":"Wed Dec 6 09:22:57 2023","interfaces":[{"interface":"back","average":{"1min":20.233000000000001,"5min":20.233000000000001,"15min":20.233000000000001},"min":{"1min":5.7640000000000002,"5min":5.7640000000000002,"15min":5.7640000000000002},"max":{"1min":121.627,"5min":121.627,"15min":121.627},"last":13.997},{"interface":"front","average":{"1min":19.085999999999999,"5min":19.085999999999999,"15min":19.085999999999999},"min":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":7.7110000000000003},"max":{"1min":74.602000000000004,"5min":74.602000000000004,"15min":74.602000000000004},"last":14.672000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":7.9130000000000003,"5min":7.9130000000000003,"15min":7.9130000000000003},"max":{"1min":71.759,"5min":71.759,"15min":71.759},"last":18.879999999999999},{"interface":"front","average":{"1min":17.98,"5min":17.98,"15min":17.98},"min":{"1min":8.8450000000000006,"5min":8.8450000000000006,"15min":8.8450000000000006},"max":{"1min":70.882999999999996,"5min":70.882999999999996,"15min":70.882999999999996},"last":11.837}]},{"osd":6,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":21.596,"5min":21.596,"15min":21.596},"min":{"1min":9.3879999999999999,"5min":9.3879999999999999,"15min":9.3879999999999999},"max":{"1min":80.668999999999997,"5min":80.668999999999997,"15min":80.668999999999997},"last":10.215999999999999},{"interface":"front","average":{"1min":23.076000000000001,"5min":23.076000000000001,"15min":23.076000000000001},"min":{"1min":11.746,"5min":11.746,"15min":11.746},"max":{"1min":72.796999999999997,"5min":72.796999999999997,"15min":72.796999999999997},"last":15.366}]},{"osd":7,"last update":"Wed Dec 6 09:22:54 2023","interfaces":[{"interface":"back","average":{"1min":29.366,"5min":29.366,"15min":29.366},"min":{"1min":10.706,"5min":10.706,"15min":10.706},"max":{"1min":207.19399999999999,"5min":207.19399999999999,"15min":207.19399999999999},"last":16.068000000000001},{"interface":"front","average":{"1min":29.623999999999999,"5min":29.623999999999999,"15min":29.623999999999999},"min":{"1min":11.667999999999999,"5min":11.667999999999999,"15min":11.667999999999999},"max":{"1min":208.68000000000001,"5min":208.68000000000001,"15min":208.68000000000001},"last":11.071999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:23:41.474 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [3, 7] 2023-12-06T09:23:41.474 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1', '3', '7'] 2023-12-06T09:23:41.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.1 3 7 2023-12-06T09:23:41.535 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:41.535 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:41.546 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:41.567 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:41.581+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:41.668 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:41.682+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:41.728 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:41.728 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:41.737 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:41.779 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:41.793+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:41.871 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:41.885+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:41.911 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:41.912 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:41.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:42.006 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:42.020+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:42.076 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:42.089+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:42.098 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:42.098 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:42.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:42.175 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:42.190+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:42.276 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:42.291+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:42.280 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:42.280 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:42.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:42.377 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:42.391+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:42.419 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:42.420 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:42.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:42.477 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:42.491+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:42.579 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:42.593+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:42.606 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:42.606 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:42.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:42.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:23:42.680 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:42.694+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:42.758 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:42.758 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:42.768 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:42.812 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:42.826+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:42.882 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:42.896+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:42.900 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:42.901 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:42.909 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:42.983 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:42.997+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:43.031 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:43.084 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:43.098+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:43.104 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:43.104 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:43.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:43.184 INFO:teuthology.orchestra.run.smithi088.stderr:set 2.1 pg_upmap mapping to [3,7] 2023-12-06T09:23:43.189 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:43.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:43.285 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:43.299+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:43.336 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:43.336 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:43.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:43.370 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:43.370 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:43.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:43.391 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:43.405+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:43.460 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:43.460 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:43.477 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:43.478 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:43.487 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:43.487 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:43.500+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:43.549 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:43.550 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:43.588 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:43.602+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:43.636 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:43.636 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:43.677 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:43.678 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:43.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:43.689 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:43.703+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:43.737 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:43.737 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:43.790 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:43.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:43.847 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:43.847 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:43.855 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:43.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:43.900 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:43.914+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:43.946 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:43.947 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:43.991 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:43.992 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:43.992 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:43.992 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:43.993 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:44.006+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:43.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:44.036 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:44.037 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:44.050 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":20,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:44.105 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:44.119+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:44.169 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:44.169 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:44.169 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:44.169 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:44.169 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:44.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:44.194 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:44.208+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:44.198 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:23:44.309 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:44.321+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 105.691875165, 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:23:44.398 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:23:44.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.997421296, 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:23:44.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:23:44.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:23:44.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.895592716, 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:23:44.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.55452882, 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:23:44.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:23:44.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.545702539, 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:23:44.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:23:44.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.545080867, 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:23:44.425 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:23:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:23:44.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:23:44.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.53713438600001, 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:23:44.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:23:44.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.436 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.536660368, 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:23:44.441 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.536206013, 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:23:44.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 104.152213762, 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:23:44.453 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 103.767064357, 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:23:44.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:23:44.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:23:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:23:44.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 103.754789046, 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.460 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:23:44.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.464 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 103.753303468, 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:23:44.465 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.466 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.467 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.846766310000007, 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.469 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.470 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.471 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:23:44.472 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.841634006999996, 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:23:44.474 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:23:44.475 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:23:44.476 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.840807976999997, 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:44.477 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:23:44.478 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:23:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.480 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.838238516000004, 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:23:44.481 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.482 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.484 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:44.485 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.837493046000006, 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.486 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.487 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:23:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:23:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.488 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:23:44.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.490 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.836686975000006, 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.491 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:23:44.492 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.493 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:23:44.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:23:44.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.496 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:23:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 95.825582522999994, 2023-12-06T09:23:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:23:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:44.497 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.498 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.499 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:23:44.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:44.501 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:44.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:44.503 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:44.517+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:44.604 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:44.618+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:44.634 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:44.634 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:44.634 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:44.634 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:44.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:44.705 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:44.719+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:44.772 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:44.772 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:44.772 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:44.772 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:44.772 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:44.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:44.824 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:44.838+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:44.907 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:44.921+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:44.952 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:44.952 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:44.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:44.952 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:44.953 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:44.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:45.008 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:45.022+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:45.093 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:45.093 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:45.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:45.108 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:45.123+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:45.209 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:45.223+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:45.229 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:45.229 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:45.229 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:45.230 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:45.230 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:45.239 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:45.327 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:45.341+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:45.411 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:45.425+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 89.822730010000001, 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:23:45.414 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:23:45.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:23:45.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:23:45.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:23:45.418 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:45.423 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:45.512 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:45.526+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:45.557 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:45.558 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:45.558 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:45.558 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:45.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:45.620 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:45.634+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:45.713 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:45.727+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:45.737 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:45.737 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:45.737 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:45.737 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:45.737 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:45.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:45.815 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:45.829+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:45.874 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:45.874 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:45.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:45.874 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:45.875 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:45.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:45.915 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:45.929+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:46.018 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:46.018 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:46.018 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:46.018 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:46.018 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:46.032+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:46.026 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:46.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:23:46.120 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:46.134+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:46.159 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:46.159 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:46.159 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:46.159 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:46.159 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:46.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:46.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:23:46.222 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:46.236+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:46.309 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:46.309 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:46.323 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:46.337+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 109.509956231, 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:46.417 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 107.09224611400001, 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:46.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:46.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 107.088061669, 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:46.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:46.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:46.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:46.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.441 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 107.002530583, 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.68855396799999, 2023-12-06T09:23:46.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:46.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:46.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:46.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:46.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.687036124, 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.453 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:46.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:46.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.68637074900001, 2023-12-06T09:23:46.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:46.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:46.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:46.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:46.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.472 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.682110577, 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:46.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:46.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:46.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:46.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:46.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:46.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.681477368, 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.479 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:46.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:46.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:46.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:46.484 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.68085026599999, 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.485 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.486 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.486 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.486 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.491 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:46.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.491 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:46.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.592590488, 2023-12-06T09:23:46.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:46.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:46.515 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 106.591669508, 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:46.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 105.839291577, 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.522 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.524 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:46.526 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.527 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.529 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.969256606000002, 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:46.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.968083094999997, 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:46.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:46.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:46.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.959931123999993, 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:46.542 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:46.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:46.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:46.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:46.548 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.949649187999995, 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.549 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:46.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:46.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.948991308000004, 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:46.554 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.555 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.555 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.555 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.555 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.555 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:46.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.561 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.948316528999996, 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.567 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 97.915958938000003, 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:46.568 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:46.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:46.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:46.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:46.574 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.574 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.574 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.574 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:46.574 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:46.574 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:46.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:46.576 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:46.577 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:46.577 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:46.577 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:46.589+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:46.641 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:46.641 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:46.676 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:46.690+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:46.732 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:46.732 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:46.779 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:46.779 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:46.779 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:46.793+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:46.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:46.834 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:46.880 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:46.894+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:46.920 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:46.920 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:46.961 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:46.961 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:46.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:47.007 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:47.008 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:47.021+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:47.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:47.102 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:47.102 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:47.102 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:47.116+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:47.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:47.184 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:47.198+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:47.285 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:47.299+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:47.294 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:47.295 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:47.304 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:47.386 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:47.400+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:47.432 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:47.432 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:47.441 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:47.488 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:47.501+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:47.530 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:07.984 94656== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:47.574 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:47.575 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:47.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:47.625 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:47.639+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:47.688 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:47.702+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:47.714 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:47.714 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:47.715 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":20,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:47.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:47.764 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:08.219 94656== Warning: unimplemented fcntl command: 1036 2023-12-06T09:23:47.789 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:47.803+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:23:47.862 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:47.862 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:47.872 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:47.892 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:47.906+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:47.993 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:48.007+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:48.048 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:48.048 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:48.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:48.099 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:48.113+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:48.201 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:48.201 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:48.201 INFO:tasks.thrashosds.thrasher:Adding osd 0 2023-12-06T09:23:48.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2023-12-06T09:23:48.225 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:48.239+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:48.239 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:48.239 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:48.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:48.299 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:48.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:48.400 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:48.414+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:48.416 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:48.416 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:48.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:48.501 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:48.515+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:48.556 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:48.556 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:48.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:48.611 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:48.625+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:48.707 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:48.721+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:48.713 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:48.713 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:48.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:48.790 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:23:48.794+0000 ff5f640 -1 mon.c@2(leader).osd e43 definitely_dead 0 2023-12-06T09:23:48.808 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:48.822+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:48.889 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:48.889 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:48.899 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:48.940 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:48.954+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:48.984 INFO:teuthology.orchestra.run.smithi088.stderr:marked in osd.0. 2023-12-06T09:23:49.002 INFO:tasks.thrashosds.thrasher:Added osd 0 2023-12-06T09:23:49.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:49.058 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:49.072+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:49.110 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:49.124+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:49.127 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:49.127 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:49.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:49.221 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:49.235+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:49.299 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:49.307 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:49.307 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:49.312 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:49.326+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:49.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:49.383 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:49.383 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:49.413 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:49.427+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:49.465 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:49.489 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:49.489 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:49.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:49.542 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:49.556+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:49.553 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:49.553 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:49.615 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:49.629+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:49.644 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:49.645 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:49.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:49.656 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:49.716 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:49.730+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:49.743 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:49.743 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:49.785 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:49.785 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:49.785 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:49.785 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:49.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:49.830 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:49.831 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:49.831 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:49.844+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:49.846 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:23:49.918 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:49.919 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:49.919 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:49.933+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:49.979 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:49.979 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:49.979 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:49.979 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:49.979 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:49.989 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:50.033 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:50.047+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:50.120 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:50.134+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:50.128 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:50.128 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:50.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:50.128 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:50.129 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:50.137 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:50.220 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:50.234+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:50.310 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:50.310 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:50.310 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:50.310 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:50.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:50.322 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:50.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:50.423 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:50.437+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:50.461 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:50.461 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:50.461 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:50.461 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:50.461 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:50.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:50.524 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:50.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:50.625 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:50.639+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:50.721 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 113.809588402, 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.722 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:23:50.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:23:50.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.730 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.391878285, 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:23:50.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:23:50.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:23:50.737 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.738 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.738 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.38769384, 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.739 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.741 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.741 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.741 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:23:50.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.741 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:23:50.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.30216275399999, 2023-12-06T09:23:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:50.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.98818613900001, 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.754 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:23:50.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:23:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:23:50.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.760 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.986668295, 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.762 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:23:50.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:23:50.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:23:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.98600292, 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:23:50.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.981742748, 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.981109539, 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:23:50.783 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.784 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:23:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:23:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:23:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:23:50.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.98048243700001, 2023-12-06T09:23:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:23:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.798 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:23:50.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:23:50.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.804 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.892222659, 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:50.810 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.89130167899999, 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:23:50.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 110.138923748, 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.816 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:23:50.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:23:50.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:50.821 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.268888777, 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:23:50.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.267715266, 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:23:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:23:50.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.25956329500001, 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.832 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:23:50.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.24928135899999, 2023-12-06T09:23:50.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.842 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.248623479, 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:23:50.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.24794869999999, 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:23:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:23:50.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:23:50.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.215591109, 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:23:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:23:50.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:23:50.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.858 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:50.859 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:50.859 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:50.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:23:50.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:50.861 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:50.874+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:50.961 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:50.976+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:51.033 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:51.034 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:51.034 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:51.034 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:51.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:51.084 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:51.098+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:51.165 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:51.179+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:51.212 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:51.212 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:51.212 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:51.212 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:51.212 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:51.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:51.266 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:51.280+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:51.367 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:51.381+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:51.395 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:23:51.396 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:51.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:51.468 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:51.482+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:51.547 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:23:51.547 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:51.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:51.560 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:23:51.569 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:51.583+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:51.574 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:23:51.574 INFO:tasks.mon_recovery:stopping all monitors 2023-12-06T09:23:51.575 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T09:23:51.575 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:23:51.598 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:23:51.599+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 94654) UID: 0 2023-12-06T09:23:51.598 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:23:51.601+0000 9f54640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-06T09:23:51.670 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:51.684+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:51.693 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:51.693 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:23:51.693 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:23:51.693 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:23:51.693 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:51.702 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:51.771 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:51.785+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:51.873 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:51.887+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:51.874 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 96.913403454000004, 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:23:51.875 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:23:51.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:23:51.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:23:51.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:23:51.879 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:23:51.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:51.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:23:51.978 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:51.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:52.021 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:52.021 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:52.021 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:52.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:52.075 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:52.089+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:52.167 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:23:52.167 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:23:52.178 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:52.178 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:52.178 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:52.179 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:52.193+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:52.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:52.281 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:52.295+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:52.334 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:12.789 94656== 2023-12-06T09:23:52.334 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:12.789 94656== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:23:52.342 INFO:tasks.ceph.mon.b.smithi132.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:23:52.356 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:23:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:23:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:23:52.357 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:52.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:52.408 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:52.422+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:52.454 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:23:52.454 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:23:52.485 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:52.498+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:52.535 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:52.535 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:52.535 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:23:52.535 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:52.544 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:23:52.548 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:23:52.586 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:52.600+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:23:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:23:52.688 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:52.702+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:52.719 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:23:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:23:52.719 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:23:52.719 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:52.724 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:23:52.725 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:23:52.731 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:52.790 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:52.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:52.834 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:23:52.850 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:23:52.880 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:52.881 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:52.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:52.892 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:52.906+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:52.922 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:23:52.922 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:23:53.005 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:53.019+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:53.009 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:23:53.010 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:23:53.020 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.020 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.030 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:53.115 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:53.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:53.201 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.202 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.205 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:53.218+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:53.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:53.295 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:53.310+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:53.378 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.379 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:53.432 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:53.446+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:53.498 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:53.512+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.525 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:53.599 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:53.613+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:53.699 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.700 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.700 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:53.715+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:53.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:53.802 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:53.817+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:53.836 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.836 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:53.903 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:53.917+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:53.980 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:53.980 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:53.989 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:54.003 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:54.003 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:54.003 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-12-06T09:23:54.003 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T09:23:54.003 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T09:23:54.004 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-12-06T09:23:54.004 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:54.018+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:54.105 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:54.119+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:23:54.136 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T09:23:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:23:54.156 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:54.156 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:54.165 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:54.207 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:54.221+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:54.308 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:54.322+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:54.335 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:54.335 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:54.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:54.409 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:54.423+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:54.473 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:54.473 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:54.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:54.524 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:54.537+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:54.610 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:54.624+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:54.652 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:54.652 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:54.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:54.712 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:54.726+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:54.793 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:54.793 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:54.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:54.846 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:54.860+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:54.914 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:54.928+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:54.928 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:54.928 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:54.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:55.015 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:55.029+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:55.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:23:55.109 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:55.109 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:55.120 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:55.121 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:55.134+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:55.221 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:55.235+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:55.247 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:55.247 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:55.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:55.321 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:55.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:55.422 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:55.437+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:55.427 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:55.427 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:55.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:55.524 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:55.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:55.609 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:55.609 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:55.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:55.624 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:55.638+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:55.727 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:55.741+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:55.749 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:55.749 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:55.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:55.827 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:55.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:55.898 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:55.899 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:55.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:55.928 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:55.942+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:56.029 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:56.043+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:56.077 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:56.078 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:56.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:23:56.130 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:56.144+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:56.231 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:56.245+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:56.255 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:56.255 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:56.265 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:23:56.332 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:56.346+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:56.399 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:56.399 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:56.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:23:56.433 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:56.448+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:56.535 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:56.548+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:56.579 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:56.580 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:56.589 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:23:56.673 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:56.687+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:56.736 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:56.750+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:56.760 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:56.760 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:56.770 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:23:56.837 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:56.852+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:56.939 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:56.939 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:56.939 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:56.953+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:56.949 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:23:57.040 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:57.054+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:57.082 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:57.083 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:57.099 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:23:57.141 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:57.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:57.230 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:57.230 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:57.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:23:57.241 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:57.255+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:57.353 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:57.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:57.367 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:57.367 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:57.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:23:57.447 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:57.461+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:57.545 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:57.546 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:57.547 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:57.561+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:57.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:23:57.648 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:57.662+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:57.676 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:23:57.676 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T09:23:57.677 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T09:23:57.677 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T09:23:57.677 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:23:57.684 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:57.685 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:57.695 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:23:57.696+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 97442) UID: 0 2023-12-06T09:23:57.695 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:23:57.697+0000 9f54640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2023-12-06T09:23:57.696 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:23:57.749 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:57.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:57.850 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:57.864+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:57.864 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:57.864 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:57.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:23:57.951 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:57.965+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:58.010 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.011 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:23:58.061 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:58.075+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:58.154 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:58.168+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:58.199 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.200 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.203 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:49.740 97447== 2023-12-06T09:23:58.203 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:49.740 97447== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:23:58.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:23:58.215 INFO:tasks.ceph.mon.c.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:23:58.255 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:58.270+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:58.341 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.341 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.350 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:23:58.356 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:58.371+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:58.458 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:58.472+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:58.477 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.477 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:23:58.559 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:58.573+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:58.617 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.618 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.628 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:23:58.669 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:23:58.683+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:23:58.760 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:58.775+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:58.797 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.797 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:23:58.891 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:58.905+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:58.963 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:58.977+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:58.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:23:59.068 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:23:59.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:23:59.146 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:23:59.147 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:23:59.147 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 2, 3, 4, 5, 6, 7, 0] 2023-12-06T09:23:59.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2023-12-06T09:23:59.151 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:59.151 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:59.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:23:59.203 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:23:59.217+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:23:59.267 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:59.281+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:23:59.363 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:59.363 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:59.368 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:23:59.382+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:23:59.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:23:59.469 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:59.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:59.536 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:59.536 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:59.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:23:59.594 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:59.604+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:59.671 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:23:59.684+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:23:59.693 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:59.693 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:59.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:23:59.771 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:23:59.785+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:23:59.844 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:23:59.844 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:23:59.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:23:59.897 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:23:59.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:23:59.973 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:23:59.987+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:00.024 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:00.024 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:00.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:00.074 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:00.088+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:00.175 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:00.188+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:00.215 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:00.215 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:00.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:00.276 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:00.290+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:00.377 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:00.392+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:00.392 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:00.393 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:00.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:00.478 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:00.492+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:00.531 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:00.531 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:00.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:00.586 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:00.600+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:00.670 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:00.670 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:00.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:00.682 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:00.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:00.782 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:00.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:00.851 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:00.851 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:00.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:00.901 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:00.915+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:00.984 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:00.998+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:01.027 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:01.028 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:01.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:01.121 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:01.135+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:01.186 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:01.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:01.206 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:01.207 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:01.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:01.308 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:01.321+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:01.390 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:01.391 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:01.391 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:01.405+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:01.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:01.492 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:01.505+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:01.571 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:01.571 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:01.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:01.627 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:01.641+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:01.703 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:01.717+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:01.716 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:01.716 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:01.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:01.795 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:01.809+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:01.870 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:01.870 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:01.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:01.924 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:01.937+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:02.003 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:02.016+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:02.008 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.008 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:02.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:02.104 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:02.118+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:02.187 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.187 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:02.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:02.205 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:02.219+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:02.307 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:02.320+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:02.323 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.324 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:02.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:02.418 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:02.431+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:02.500 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.501 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:02.508 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:02.522+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:02.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:02.609 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:02.622+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.684 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:02.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:02.710 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:02.724+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:02.811 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:02.825+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:02.859 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.859 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:02.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:02.912 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:02.926+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:02.994 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:02.994 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:03.050 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:03.062+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:03.115 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:03.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:03.134 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:03.134 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:03.228 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:03.242+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:03.313 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:03.313 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.315 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:03.329+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:03.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:03.416 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:03.430+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:03.493 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:03.493 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.503 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:03.544 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:03.558+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:03.618 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:03.632+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:03.673 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:03.673 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:03.724 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:03.738+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:24:03.779 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T09:24:03.779 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T09:24:03.780 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T09:24:03.780 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:24:03.799 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:24:03.798+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 94328) UID: 0 2023-12-06T09:24:03.799 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:24:03.799+0000 9f54640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2023-12-06T09:24:03.820 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:03.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:03.820 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:03.820 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:03.921 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:03.935+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:03.958 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:03.958 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:03.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:04.055 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:04.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:04.124 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:04.137+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:04.143 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:04.143 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:04.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:04.237 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:04.251+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:04.323 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:04.323 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:04.328 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:04.342+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:04.332 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:04.367 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:28.038 94333== 2023-12-06T09:24:04.367 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:28.038 94333== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:24:04.377 INFO:tasks.ceph.mon.a.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:24:04.429 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:04.443+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:04.504 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:04.505 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:04.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:04.559 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:04.573+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:04.631 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:04.645+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:04.646 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:04.646 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:04.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:04.742 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:04.756+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:04.830 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:04.831 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:04.836 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:04.848+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:04.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:04.935 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:04.948+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:05.009 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.009 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:05.060 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:05.073+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:05.136 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:05.150+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:05.186 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.186 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:05.239 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:05.253+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:05.322 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.323 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.332 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:05.373 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:05.387+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:05.439 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:05.453+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:05.500 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.501 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:05.555 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:05.569+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:05.637 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.637 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.640 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:05.654+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:05.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:05.741 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:05.755+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:05.782 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.782 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:05.842 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:05.856+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:05.944 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:05.958+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:05.961 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:05.961 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:05.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:06.055 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:06.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:06.139 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:06.139 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:06.146 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:06.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:06.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:06.247 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:06.261+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:06.316 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:06.316 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:06.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:06.348 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:06.362+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:06.453 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:06.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:06.496 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:06.496 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:06.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:06.554 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:06.568+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:06.633 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:06.633 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:06.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:06.654 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:06.668+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:06.759 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:06.773+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:06.774 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:06.775 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:06.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:06.869 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:06.883+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:06.955 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:06.955 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:06.958 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:06.972+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:06.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:07.059 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:07.073+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:07.138 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:07.138 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:07.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:07.194 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:07.208+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:07.261 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:07.275+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:07.282 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:07.282 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:07.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:07.379 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:07.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:07.460 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:07.460 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:07.463 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:07.477+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:07.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:07.567 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:07.580+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:07.638 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:07.639 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:07.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:07.694 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:07.708+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:07.768 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:07.782+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:07.795 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:07.795 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:07.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:07.869 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:07.883+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:07.973 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:07.986+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:07.974 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:07.974 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:07.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:08.073 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:08.087+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:08.152 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:08.152 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:08.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:08.206 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:08.220+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:08.276 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:08.290+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:08.292 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:08.292 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:08.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:08.386 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:08.399+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:08.468 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:08.468 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:08.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:08.478 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:08.492+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:08.579 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:08.593+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:08.647 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:08.647 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:08.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:08.698 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:08.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:08.781 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:08.795+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:08.826 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:08.827 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:08.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:08.920 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:08.934+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:08.983 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:08.997+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:09.004 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:09.004 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:09.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:09.084 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:09.098+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:09.180 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:09.180 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:09.185 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:09.199+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:09.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:09.286 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:09.300+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:09.360 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:09.360 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:09.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:09.414 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:09.428+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:09.487 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:09.501+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:09.500 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:09.500 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:09.509 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:09.593 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:09.607+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:09.677 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:09.677 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:09.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:09.688 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:09.702+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:09.790 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:09.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:09.826 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:09.827 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:09.836 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:24:09.882 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T09:24:09.882 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T09:24:09.882 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2023-12-06T09:24:09.882 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T09:24:09.882 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T09:24:09.886 INFO:tasks.ceph.mon.b:Started 2023-12-06T09:24:09.887 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T09:24:09.887 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T09:24:09.890 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:09.904+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:09.921 INFO:tasks.ceph.mon.c:Started 2023-12-06T09:24:09.921 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:24:09.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:24:09.992 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:10.006+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:10.022 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:10.022 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:10.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:10.092 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:10.106+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:10.164 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:10.164 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:10.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:10.218 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:10.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:10.294 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:10.308+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:10.306 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:10.306 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:10.317 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:10.401 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:10.415+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:10.487 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:10.488 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:10.495 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:10.509+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:10.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:10.596 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:10.610+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:10.666 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:10.666 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:10.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:10.697 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:10.711+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:10.798 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:10.812+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:10.823 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:10.823 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:10.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:10.906 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:10.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:10.999 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:11.013+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:11.021 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:11.022 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:11.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:11.101 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:11.115+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:11.161 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:11.161 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:11.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:11.212 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:11.226+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:11.303 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:11.317+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:11.344 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:11.344 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:11.354 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:11.438 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:11.452+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:11.505 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:11.519+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:11.528 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:11.529 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:11.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:11.606 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:11.620+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:11.707 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:11.721+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:11.710 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:11.710 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:11.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:11.807 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:11.821+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:11.851 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:11.852 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:11.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:11.946 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:11.960+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:12.009 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:12.023+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:12.033 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:12.033 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:12.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:12.110 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:12.124+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:12.211 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:12.224+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:12.215 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:12.215 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:12.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:12.311 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:12.325+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:12.404 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:12.404 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:12.413 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:12.426+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:12.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:12.514 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:12.528+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:12.593 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:12.636 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:12.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:12.715 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:12.729+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:12.725 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:12.725 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:12.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:12.817 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:12.831+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:12.906 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:12.906 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:12.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:12.957 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:12.971+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:13.020 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:13.034+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:13.086 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:13.086 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:13.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:13.142 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:13.156+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:13.221 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:13.235+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:13.231 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:13.231 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:13.241 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:13.322 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:13.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:13.415 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:13.415 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:13.424 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:13.437+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:13.425 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:13.525 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:13.539+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:13.597 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:13.597 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:13.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:13.650 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:13.664+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:13.727 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:13.741+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:13.735 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:13.735 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:13.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:13.828 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:13.842+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:13.878 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:13.878 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:13.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:13.929 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:13.942+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:14.030 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:14.045+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:14.132 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:14.146+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:14.161 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.162 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:14.233 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:14.247+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:14.300 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.300 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:14.351 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:14.365+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:14.445 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:14.458+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:14.548 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:14.561+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:14.672 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:14.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:14.749 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:14.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:14.802 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.802 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.813 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:14.857 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:14.871+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:14.941 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:14.941 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:14.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:14.952 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:14.965+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:15.052 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:15.066+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:15.121 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:15.121 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:15.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:15.153 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:15.167+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:15.254 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:15.268+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:15.301 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:15.301 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:15.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:15.355 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:15.368+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:15.441 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:15.441 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:15.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:15.455 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:15.469+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:15.556 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:15.570+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:15.599 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:15.599 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:15.608 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:15.657 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:15.671+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:15.740 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:15.740 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:15.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:15.758 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:15.772+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:15.859 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:15.873+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:15.918 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:15.919 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:15.928 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:15.969 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:15.983+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:16.061 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:16.075+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:16.097 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:16.097 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:16.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:16.162 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:16.176+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:16.263 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:16.277+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:16.276 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:16.277 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:16.287 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:16.372 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:16.385+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:16.461 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:16.462 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:16.465 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:16.478+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:16.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:16.566 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:16.580+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:16.643 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:16.643 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:16.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:16.698 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:16.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:16.770 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:16.784+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:16.782 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:16.783 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:16.792 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:16.878 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:16.891+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:16.963 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:16.963 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:16.972 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:16.986+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:16.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:17.073 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:17.087+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:17.146 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:17.146 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:17.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:17.199 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:17.213+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:17.275 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:17.289+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:17.285 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:17.285 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:17.296 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:17.376 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:17.390+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:17.465 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:17.465 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:17.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:17.476 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:17.490+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:17.577 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:17.591+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:17.606 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:17.606 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:17.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:17.678 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:17.692+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:17.753 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:17.754 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:17.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:17.807 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:17.821+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:17.866 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:07.920 96574== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:17.880 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:17.894+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:17.891 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:17.891 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:17.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:17.985 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.001 102760== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:17.985 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:17.999+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:18.070 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:18.070 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:18.075 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:08.129 96574== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:18.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:18.122 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:18.136+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:18.183 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:18.197+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:18.229 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.245 102760== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:18.251 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:18.251 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:18.260 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:18.284 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:18.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:18.385 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:18.399+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:18.432 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:18.432 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:18.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:18.485 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:18.499+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:18.586 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:18.600+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:18.611 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:18.611 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:18.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:18.686 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:18.700+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:18.787 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:18.801+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:18.798 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:18.798 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:18.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:18.902 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:18.916+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:18.985 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:18.985 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:18.991 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:19.006+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:18.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:19.092 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:19.106+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:19.162 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:19.162 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:19.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:19.215 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:19.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:19.294 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:19.308+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:19.300 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:19.300 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:19.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:19.401 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:19.414+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:19.480 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:19.481 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:19.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:19.532 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:19.546+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:19.596 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:19.611+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:19.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:19.721 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:19.735+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:19.799 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:19.812+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:19.805 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:19.805 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:19.815 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:19.900 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:19.915+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:19.986 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:19.986 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:19.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:20.000 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:20.014+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:20.101 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:20.115+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:20.165 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:20.165 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:20.174 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:20.218 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:20.233+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:20.304 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:20.318+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:20.327 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:20.328 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:20.337 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:20.405 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:20.419+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:20.506 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:20.520+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:20.506 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:20.506 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:20.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:20.607 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:20.621+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:20.648 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:20.648 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:20.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:20.708 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:20.722+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:20.800 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:20.800 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:20.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:20.810 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:20.824+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:20.910 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:20.924+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:20.985 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:20.985 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:20.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:21.011 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:21.025+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:21.112 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:21.127+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:21.134 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:21.134 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:21.139 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:21.213 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:21.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:21.288 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:21.288 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:21.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:21.314 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:21.328+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:21.415 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:21.429+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:21.435 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:21.435 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:21.447 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:21.517 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:21.530+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:21.581 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:21.581 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:21.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:21.641 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:21.655+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:21.719 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:21.733+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:21.728 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:21.728 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:21.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:21.826 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:21.840+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:21.912 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:21.912 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:21.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:21.922 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:21.935+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:22.023 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:22.037+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:22.092 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:22.093 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:22.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:22.124 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:22.138+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:22.225 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:22.239+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:22.272 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:22.272 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:22.282 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:22.325 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:22.339+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:22.426 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:22.440+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:22.449 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:22.449 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:22.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:22.545 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:22.558+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:22.628 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:22.628 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:22.629 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:22.643+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:22.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:22.730 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:22.744+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:22.811 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:22.811 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:22.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:22.862 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:22.876+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:22.932 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:22.946+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:22.988 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:22.989 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:22.998 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:23.039 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:23.053+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:23.135 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:23.148+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:23.166 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:23.166 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:23.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:23.236 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:23.250+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:23.304 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:23.304 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:23.313 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:23.357 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:23.371+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:23.442 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:23.442 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:23.442 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:23.452+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:23.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:23.541 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:23.553+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:23.620 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:23.620 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:23.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:23.674 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:23.688+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:23.741 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:23.755+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:23.768 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:23.768 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:23.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:23.842 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:23.856+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:23.944 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:23.958+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:23.947 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:23.947 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:23.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:24.046 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:24.060+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:24.088 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:24.088 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:24.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:24.147 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:24.161+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:24.249 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:24.262+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:24.268 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:24.268 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:24.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:24.349 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:24.363+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:24.403 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:24.404 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:24.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:24.450 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:24.464+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:24.541 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:24.541 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:24.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:24.551 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:24.564+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:24.653 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:24.667+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:24.722 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:24.722 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:24.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:24.776 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:24.790+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:24.857 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:24.871+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:24.867 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:24.867 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:24.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:24.958 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:24.972+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:25.041 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:25.041 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:25.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:25.093 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:25.107+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:25.160 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:25.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:25.220 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:25.220 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:25.230 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:25.261 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:25.275+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:25.362 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:25.375+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:25.401 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:25.401 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:25.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:25.463 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:25.476+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:25.532 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:25.533 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:25.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:25.587 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:25.601+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:25.665 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:25.679+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:25.683 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:25.683 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:25.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:25.768 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:25.780+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:25.858 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:25.858 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:25.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:25.868 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:25.881+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:25.968 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:25.982+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:26.035 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:26.035 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:26.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:26.069 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:26.083+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:26.175 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:26.189+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:26.212 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:26.212 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:26.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:26.276 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:26.290+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:26.349 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:26.349 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:26.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:26.377 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:26.391+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:26.478 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:26.492+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:26.531 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:26.531 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:26.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:26.584 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:26.598+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:26.668 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:26.668 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:26.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:26.719 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:26.732+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:26.781 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:26.795+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:26.849 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:26.849 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:26.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:26.896 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:26.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:26.984 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:26.998+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:27.038 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:27.038 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:27.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:27.085 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:27.099+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:27.181 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:27.181 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:27.185 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:27.199+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:27.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:27.287 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:27.300+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:27.361 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:27.361 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:27.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:27.412 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:27.426+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:27.488 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:27.502+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:27.551 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:27.551 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:27.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:27.605 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:27.619+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:27.689 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:27.703+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:27.690 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:27.690 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:27.699 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:27.791 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:27.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:27.870 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:27.870 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:27.879 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:27.890 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:24:27.805+0000 ff5f640 -1 mon.c@2(leader).osd e45 definitely_dead 0 2023-12-06T09:24:27.891 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:27.905+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:27.994 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:28.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:28.010 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:28.010 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:28.020 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:28.086 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:28.104 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:28.118+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:28.177 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:28.188 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:28.188 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:28.194 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:28.207+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:28.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:28.277 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:28.277 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:28.296 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:28.310+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:28.364 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:28.365 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:28.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:28.378 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:28.378 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:28.415 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:28.429+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:28.466 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:28.497 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:28.511+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:28.578 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:28.579 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:28.598 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:28.612+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:28.623 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:24:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.623 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.91729551500001, 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.624 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:24:28.625 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.626 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.22284164600001, 2023-12-06T09:24:28.628 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:24:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:24:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.631 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:24:28.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:24:28.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.12101306599999, 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033035596, 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.635 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476370+0000", 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.476379+0000", 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.6170000000000007e-06 2023-12-06T09:24:28.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477578+0000", 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001199019 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477720+0000", 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014209699999999999 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.477832+0000", 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011230699999999999 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482308+0000", 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044757199999999999 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.482689+0000", 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038125399999999998 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.508778+0000", 2023-12-06T09:24:28.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026089186 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.509406+0000", 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00062739600000000001 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.77994917000001, 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:24:28.639 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:24:28.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:24:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:24:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.644 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.771122889, 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:28.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:24:28.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:24:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:24:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.648 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.770501217, 2023-12-06T09:24:28.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.650 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.653 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.762554736, 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.76208071799999, 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:24:28.658 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:24:28.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:24:28.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.761626363, 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.662 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:24:28.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:24:28.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 148.37763411200001, 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.666 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.99248470699999, 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.670 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.98020939599999, 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:24:28.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 147.97872381799999, 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:24:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:24:28.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:24:28.681 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.07218666, 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:24:28.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:24:28.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:24:28.685 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.06705435699999, 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:24:28.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:24:28.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:24:28.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.066228327, 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:28.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:24:28.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:24:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:24:28.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.063658866, 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:24:28.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.062913396, 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:24:28.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.701 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.062107325, 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:24:28.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:24:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 140.05100287299999, 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:24:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:24:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:28.710 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:28.710 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:28.711 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:28.712 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:28.712 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:28.725+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:28.761 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:24:28.761 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:24:28.800 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:24:28.813 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:28.827+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:28.882 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:28.882 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:28.882 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:28.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:28.936 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:28.950+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:28.997 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:24:29.012 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:24:29.013 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T09:24:29.013 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T09:24:29.015 INFO:tasks.ceph.mon.a:Started 2023-12-06T09:24:29.015 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:24:29.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:24:29.017 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:29.031+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:29.070 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:29.071 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:29.071 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:29.071 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:29.071 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:29.080 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:29.094 INFO:teuthology.orchestra.run.smithi088.stderr:marked out osd.5. 2023-12-06T09:24:29.124 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:29.138+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:29.216 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:29.216 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:29.216 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:29.216 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:24:29.216 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:29.219 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:29.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:29.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:29.323 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:29.337+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:29.396 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:29.396 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:29.396 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:29.396 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:29.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:29.424 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:29.438+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:29.525 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:29.539+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:29.573 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:29.574 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:29.574 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:29.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:29.626 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:29.640+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:29.686 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 134.12166186100001, 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.093125244999999995, 2023-12-06T09:24:29.713 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 2 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:29.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.603914+0000", 2023-12-06T09:24:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.604131+0000", 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021746499999999999 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.608705+0000", 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0045735480000000002 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.612768+0000", 2023-12-06T09:24:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0040635799999999998 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.621519+0000", 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0087511180000000004 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.642371+0000", 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020851833 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:15.697039+0000", 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054667700999999999 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:29.720 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:29.728 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:29.741 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:29.755+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:29.830 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:29.844+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:29.873 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:29.873 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:29.873 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:29.873 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:29.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:29.931 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:29.944+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:30.032 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:30.046+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:30.067 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:30.067 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:30.067 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:30.067 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:30.068 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:30.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:30.139 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:30.151+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:30.178 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:20.194 102760== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:30.230 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:24:30.230 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:30.234 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:30.248+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:30.239 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:30.252 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:20.306 96574== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:30.263 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:20.276 102760== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:30.307 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:20.362 96574== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:30.335 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:30.349+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:30.437 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:30.450+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:30.450 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:30.450 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:30.451 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:30.451 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:30.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:30.490 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:20.545 96574== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:30.534 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:20.551 102760== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:30.632 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:30.614+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:30.670 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:30.670 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:30.670 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:30.670 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:30.670 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:30.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:30.768 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:30.782+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:30.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:24:30.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:24:30.841 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:30.855+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:30.859 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:30.865+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:30.942 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:30.956+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:31.003 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:31.003 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:31.003 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:31.003 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 154.08590536299999, 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.004 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.005 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.006 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.007 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.008 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.009 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:31.010 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.66819524600001, 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.011 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.012 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.013 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.014 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.66401080099999, 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:24:31.019 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:24:31.019 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.019 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.019 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.019 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:24:31.019 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:24:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:24:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:24:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:24:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.024 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.57847971499999, 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:31.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.026 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:24:31.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:24:31.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:24:31.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:24:31.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.26450310000001, 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:24:31.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:24:31.038 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.038 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.038 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.26298525600001, 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.039 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:24:31.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:24:31.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:24:31.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.045 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.262319881, 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:24:31.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:24:31.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:24:31.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:24:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.053 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.25805970900001, 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:24:31.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:24:31.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:24:31.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:24:31.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.25742650000001, 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.061 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:24:31.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:24:31.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.068 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.256799398, 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:24:31.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:24:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:24:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:24:31.074 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.075 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.075 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.075 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.075 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.16853961999999, 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:24:31.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.081 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 151.16761864, 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:31.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:24:31.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:24:31.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:24:31.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 150.41524070899999, 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:24:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:24:31.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:24:31.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:24:31.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:24:31.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:24:31.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:24:31.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.54520573799999, 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.095 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:24:31.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:24:31.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:24:31.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.544032227, 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:24:31.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:24:31.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:24:31.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:24:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.53588025600001, 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.109 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:24:31.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:24:31.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.52559832, 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:31.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:24:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:24:31.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.52494043999999, 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:24:31.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.52426566099999, 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.130 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:24:31.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:24:31.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.135 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.136 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 142.49190806999999, 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:31.137 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:31.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:24:31.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:24:31.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:31.143 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:31.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:31.147 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:31.147 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:31.147 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:31.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:31.219 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:31.220 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:31.247 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:31.261+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:31.328 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:31.350 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:31.364+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:31.383 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:31.383 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:31.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:31.430 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:31.431 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:31.454 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:31.465+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:31.538 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:31.538 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:31.552 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:31.566+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:31.564 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:31.564 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:31.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:31.625 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:31.626 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:31.653 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:31.667+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:31.704 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:31.704 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:31.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:31.722 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:31.770 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:31.784+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:31.819 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:24:31.820 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:24:31.857 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:31.858 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:31.858 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:31.872+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:31.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:31.956 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:31.970+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:32.047 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:32.047 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:32.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:32.057 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:32.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:32.157 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:32.171+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:32.226 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:32.226 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:32.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:32.258 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:32.272+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:32.360 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:32.374+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:32.366 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:32.367 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:32.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:24:32.471 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:32.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:32.543 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:32.544 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:32.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:32.562 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:32.576+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:32.663 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:32.677+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:32.681 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:32.681 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:32.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:32.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:24:32.776 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:32.790+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:32.865 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:32.879+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:32.873 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:32.873 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:32.883 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:32.968 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:32.982+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:33.052 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:33.052 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:33.061 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:33.102 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:33.116+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:33.170 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:33.183+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:33.234 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:33.234 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:33.244 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:33.289 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:33.303+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:33.371 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:33.385+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:33.374 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:24:33.375 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:33.375 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:33.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:33.473 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:33.486+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:33.553 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:33.553 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:33.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:33.604 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:33.618+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:33.674 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:33.688+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:33.729 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:33.729 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:33.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:33.780 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:33.793+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:33.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:24:33.876 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:33.890+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:33.909 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:33.909 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:33.919 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:33.977 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:33.991+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:34.074 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:34.075 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:34.078 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:34.092+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:34.088 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:34.089 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:34.097 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:34.113 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:24:34.113 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:24:34.113 INFO:tasks.thrashosds.thrasher:Adding osd 5 2023-12-06T09:24:34.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2023-12-06T09:24:34.162 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:34.162 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:34.183 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:34.196+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:34.251 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:34.251 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:34.279 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:34.293+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:34.284 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:34.284 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:34.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:34.343 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:34.343 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:34.382 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:34.396+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:34.428 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:34.428 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:34.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:24:34.437 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:34.483 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:34.497+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:34.584 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:34.598+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:34.620 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:34.621 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:34.621 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:34.621 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:34.621 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:34.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:34.670 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:34.670 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:34.685 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:34.699+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:34.772 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:34.772 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:34.777 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:34.778 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:34.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:34.778 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:24:34.778 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:34.785 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:34.799+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:34.787 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:34.859 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:34.859 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:34.886 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:34.900+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:34.945 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:24:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:24:34.961 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:34.961 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:34.961 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:34.961 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:34.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:35.013 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:24:35.000+0000 ff5f640 -1 mon.c@2(leader).osd e48 definitely_dead 0 2023-12-06T09:24:35.013 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:35.027+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:35.088 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:35.102+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:35.163 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:35.163 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:35.163 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:35.163 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:35.163 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:35.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:35.190 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:35.203+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:35.290 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:35.304+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:35.357 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:35.357 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:35.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.45184628199999, 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:24:35.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:24:35.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:24:35.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 156.03413616500001, 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.364 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:24:35.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 156.02995172000001, 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.369 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:24:35.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:24:35.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.94442063400001, 2023-12-06T09:24:35.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:24:35.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:24:35.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:24:35.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.63044401900001, 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:35.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:35.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:24:35.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:24:35.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:24:35.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:24:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:24:35.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.628926175, 2023-12-06T09:24:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:24:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:24:35.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:24:35.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:24:35.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:24:35.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:24:35.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:24:35.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.401 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.62826079999999, 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.402 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:35.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:24:35.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:24:35.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:24:35.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.624000628, 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.410 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:24:35.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:24:35.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.623367419, 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.415 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.62274031699999, 2023-12-06T09:24:35.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:24:35.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.53448053899999, 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.426 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:35.431 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 155.533559559, 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:24:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.438 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 154.78118162800001, 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:35.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:24:35.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:24:35.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.443 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.91114665699999, 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.909973146, 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.449 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:24:35.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:24:35.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.90182117500001, 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.454 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.459 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.891539239, 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:35.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:24:35.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:24:35.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:24:35.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.464 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.89088135899999, 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.465 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:24:35.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:24:35.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:35.470 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.89020658000001, 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 146.85784898899999, 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:35.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:24:35.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:24:35.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.480 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.481 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.481 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.481 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:35.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:35.483 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:35.496+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:35.583 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:35.597+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:35.605 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:24:35.616 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:35.616 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:35.616 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:35.616 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:35.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:marked in osd.5. 2023-12-06T09:24:35.682 INFO:tasks.thrashosds.thrasher:Added osd 5 2023-12-06T09:24:35.724 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:35.738+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:35.810 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:35.810 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:35.810 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:35.810 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:35.810 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:35.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:35.822 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:35.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:35.885 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:35.899+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:35.965 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:35.965 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 5.3209996049999999, 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:24:35.966 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:24:35.967 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:24:35.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.969 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 5.0157783040000004, 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:35.970 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.971 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.972 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.973 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.974 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.974 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:35.974 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:35.974 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:35.975 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:36.016 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:36.030+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:36.087 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:36.101+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:36.145 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:36.145 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:36.145 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:36.145 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:36.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:36.198 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:36.212+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:36.284 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:36.284 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:36.291 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:36.305+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:36.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:36.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:24:36.450 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:36.464+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:36.466 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 141.50505393899999, 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:24:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:36.471 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:36.471 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:36.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:36.522 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:36.536+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:36.594 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:36.608+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:36.611 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:36.611 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:36.611 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:36.611 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:36.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:36.695 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:36.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:36.787 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:36.787 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:36.787 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:36.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:36.796 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:36.810+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:36.897 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:36.911+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:36.936 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:36.936 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:36.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:36.936 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:24:36.936 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:36.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:36.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:24:37.005 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:37.019+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:37.077 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:24:37.137 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:37.151+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:37.220 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:37.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:37.245 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:37.246 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:37.246 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:37.246 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:37.246 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:37.246 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:37.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:37.353 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:37.354 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:37.363 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:37.376+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:37.408 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:37.422+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:37.467 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:37.467 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:37.467 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:37.468 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:37.468 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:37.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:37.484 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:37.484 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:37.517 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:37.531+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:37.576 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:37.577 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:37.604 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:37.618+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:37.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:24:37.664 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:37.664 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:37.687 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:37.687 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 158.98039989599999, 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:37.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:24:37.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:24:37.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:24:37.691 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 158.28594602699999, 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:24:37.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:24:37.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.843053551, 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:24:37.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.83422727000001, 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.705 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:24:37.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:24:37.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:37.710 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.83360559799999, 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:24:37.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.82565911699999, 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:24:37.715 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:24:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:24:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:24:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:24:37.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.721 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.82518509900001, 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:24:37.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:24:37.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:24:37.726 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.82473074399999, 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:24:37.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:24:37.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:24:37.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:24:37.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.440738493, 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.733 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:24:37.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.737 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.055589088, 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:24:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:24:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:24:37.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:24:37.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.743 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.04331377700001, 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:24:37.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:24:37.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:24:37.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 157.04182819900001, 2023-12-06T09:24:37.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:24:37.751 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:24:37.752 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.753 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:24:37.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:24:37.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.13529104099999, 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:24:37.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.759 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:24:37.759 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:24:37.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.759 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.759 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.760 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.13015873800001, 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.761 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:24:37.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:24:37.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:24:37.766 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.12933270799999, 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:37.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:24:37.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.12676324700001, 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:24:37.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:37.776 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.12601777699999, 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:37.780 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.12521170599999, 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.785 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 149.114107254, 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:37.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.788 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:24:37.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:24:37.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 0.34077511700000002, 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:37.793 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:37.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.796 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:37.797 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:37.797 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:37.797 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.641 104038== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:37.798 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:37.798 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:37.799 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:37.799 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:37.812+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:37.858 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:37.859 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:37.902 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:37.913+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:37.946 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:24:37.947 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:24:37.960 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.882 104038== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:37.971 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:37.971 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:37.980 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:38.022 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:38.035+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:38.106 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:38.119+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:38.153 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:38.153 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:38.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:38.201 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:38.215+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:38.303 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:38.317+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:38.331 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:38.331 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:38.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:38.404 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:38.418+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:38.469 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:38.469 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:38.478 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:24:38.505 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:38.519+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:38.606 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:38.620+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:38.607 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:38.607 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:38.617 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:24:38.708 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:38.722+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:38.784 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:38.784 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:38.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:38.835 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:38.849+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:38.910 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:38.924+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:38.926 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:38.926 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:38.935 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:39.012 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:39.026+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:39.064 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:39.065 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:39.074 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:39.117 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:39.130+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:39.185 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:24:39.213 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:39.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:39.263 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:39.263 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:39.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:39.317 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:39.331+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:39.402 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:39.402 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:39.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:39.454 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:39.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:39.519 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:39.533+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:39.584 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:39.584 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:39.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:39.639 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:39.653+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:39.721 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:39.735+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:39.736 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:39.737 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:39.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:39.834 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:39.846+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:39.919 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:39.920 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:39.922 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:39.936+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:39.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:39.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:24:40.023 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:40.037+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:40.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:24:40.119 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:40.119 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:40.124 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:40.138+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:40.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:40.264 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:40.276+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:40.326 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:40.340+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:40.334 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:40.335 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:40.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:40.360 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:40.361 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:40.426 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:40.440+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:40.485 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:40.485 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:40.492 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:40.492 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:40.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:40.527 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:40.541+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:40.613 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:40.614 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:40.631 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:40.645+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:40.683 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:24:40.683 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:24:40.683 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:24:40.684 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-06T09:24:40.684 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:24:40.692 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:40.699+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86679) UID: 0 2023-12-06T09:24:40.695 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:24:40.699+0000 ac7d640 -1 osd.7 51 *** Got signal Terminated *** 2023-12-06T09:24:40.733 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:40.734 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:40.748+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:40.735 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:40.735 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:40.735 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:40.735 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:40.735 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:40.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:40.822 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:24:40.827 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:40.827 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:40.832 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:40.846+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:40.917 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:40.918 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:40.934 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:40.947+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 145.99693384099999, 2023-12-06T09:24:40.960 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.961 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:24:40.962 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:24:40.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:40.964 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:40.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:41.000 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:41.037 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:41.049+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:41.089 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:24:41.089 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:24:41.111 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:41.111 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:41.111 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:41.111 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:41.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:41.161 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:41.175+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:41.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:24:41.236 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:41.250+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:41.286 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:41.286 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:41.286 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:41.286 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:41.286 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:41.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:41.328 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:12.250 104038== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:41.336 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:41.351+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:41.397 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:12.319 104038== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:41.437 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:41.451+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:41.461 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:41.462 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:41.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:41.462 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:24:41.462 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:41.472 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:41.538 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:41.552+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:41.600 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:41.600 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:41.600 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:41.600 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:41.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:41.626 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:12.548 104038== Warning: unimplemented fcntl command: 1036 2023-12-06T09:24:41.651 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:41.665+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:41.740 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:41.754+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:41.780 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:41.781 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:41.781 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:41.781 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:41.781 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:41.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:41.843 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:41.857+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:41.943 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:41.957+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 163.279265342, 2023-12-06T09:24:41.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:41.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:41.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.584811473, 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:41.995 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:41.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:24:41.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:24:41.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.141918997, 2023-12-06T09:24:41.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:24:42.005 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.005 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.006 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.016 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.13309271599999, 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:24:42.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:24:42.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:24:42.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.132471044, 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:24:42.021 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.12452456299999, 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:24:42.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:24:42.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.12405054499999, 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.040 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 162.12359619, 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:24:42.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:24:42.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.046 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 161.73960393900001, 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:24:42.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:24:42.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:24:42.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 161.35445453400001, 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:24:42.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:24:42.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 161.34217922299999, 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:24:42.056 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:24:42.060 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 161.34069364499999, 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:24:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:24:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:24:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:24:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.434156487, 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.067 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:24:42.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:24:42.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.42902418400001, 2023-12-06T09:24:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.428198154, 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.082 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:24:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.42562869299999, 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:24:42.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:24:42.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.42488322299999, 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.102 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:24:42.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:24:42.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:24:42.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.424077152, 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:24:42.108 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:42.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:24:42.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:24:42.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 153.41297270000001, 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:24:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.119 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 4.6396405630000004, 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:42.120 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:24:42.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:42.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:24:42.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:24:42.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:42.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:42.127 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:42.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:42.130 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:42.143+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:42.230 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:42.244+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:42.257 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:42.257 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:42.257 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:42.257 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:42.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:42.331 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:42.345+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:42.406 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:42.407 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:42.407 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:42.407 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:42.407 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:42.417 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:42.458 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:42.472+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:42.533 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:42.547+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:42.589 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:42.589 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:42.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:42.590 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:24:42.590 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:42.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:42.643 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:42.657+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:42.731 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:42.731 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:42.731 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:42.731 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:42.736 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:42.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:42.740 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:42.837 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:42.851+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:42.867 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:24:42.916 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:42.916 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:42.916 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:42.917 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:42.917 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:42.926 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:42.968 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:42.981+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:43.045 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:43.054+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:43.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:24:43.132 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:43.133 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:43.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:43.133 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:24:43.133 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:43.143 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:43.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:43.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:43.240 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:43.254+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:43.318 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:43.318 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:43.318 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:43.318 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:43.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:43.340 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:43.340 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:43.344 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:43.357+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:43.428 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:43.457 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:43.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:43.470 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:43.470 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:43.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:43.516 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:43.516 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:43.544 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:43.558+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:43.605 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:43.605 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:43.645 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:43.645 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:43.645 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:43.659+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:43.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:43.688 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:43.688 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:43.746 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:43.760+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:43.772 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:43.785 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:43.785 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:43.796 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:43.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:24:43.848 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:43.861+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:43.869 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:43.948 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:43.962+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:43.952 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:24:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:24:43.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:24:43.998 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:43.998 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:44.050 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:44.064+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:44.151 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:44.164+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:44.179 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:44.180 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:44.252 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:44.266+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:44.353 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:44.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:44.361 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:44.362 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:44.454 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:44.468+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:44.504 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:44.504 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:44.532 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:24:44.546 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:24:44.546 INFO:tasks.mon_recovery:stopping all monitors 2023-12-06T09:24:44.546 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T09:24:44.546 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:24:44.560 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:24:44.564+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 96572) UID: 0 2023-12-06T09:24:44.561 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:24:44.566+0000 9f54640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-06T09:24:44.587 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:44.601+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:44.653 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:44.654 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.655 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:44.669+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:44.664 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:44.756 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:44.770+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:44.793 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:44.793 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.803 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:44.862 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:44.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:44.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:24:44.949 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:44.949 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:44.958 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:44.972+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:44.959 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:45.039 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:35.094 96574== 2023-12-06T09:24:45.040 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:35.094 96574== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:24:45.050 INFO:tasks.ceph.mon.b.smithi132.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:24:45.098 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:45.112+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:45.130 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:45.130 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:45.140 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:45.181 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:45.195+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:45.260 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:45.274+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:45.309 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:45.309 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:45.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:45.362 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:45.375+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:45.446 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:45.447 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:45.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:45.501 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:45.515+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:45.563 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:45.577+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:45.585 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:45.585 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:45.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:45.678 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:45.692+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:45.762 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:45.762 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:45.765 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:45.779+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:45.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:45.866 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:45.880+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:45.938 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:45.939 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:45.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:45.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:24:46.001 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:46.015+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:46.068 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:46.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:46.119 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:46.119 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:46.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:46.173 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:46.187+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:46.202 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:46.202 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:46.260 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:46.260 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:46.260 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:46.260 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:46.269 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:46.270 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:46.284+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:46.288 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:46.289 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:46.370 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:46.384+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:46.375 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:46.375 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:46.423 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:46.423 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:46.423 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:46.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:46.466 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:46.467 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:46.471 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:46.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:46.553 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:46.554 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:46.562 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:24:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:46.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:46.612 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:46.626+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:46.639 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:46.675 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:46.688+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:46.741 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:24:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:24:46.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:24:46.758 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:46.758 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:46.758 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:46.758 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:46.767 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:46.777 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:46.790+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:46.785 INFO:tasks.ceph.osd.7:Stopped 2023-12-06T09:24:46.785 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2023-12-06T09:24:46.785 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:24:46.785 DEBUG:teuthology.orchestra.run.smithi132:> type ceph-objectstore-tool 2023-12-06T09:24:46.806 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-06T09:24:46.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2023-12-06T09:24:46.876 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:46.890+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:46.955 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:46.955 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:46.955 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:46.955 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:46.955 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:46.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:47.007 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:47.021+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:47.079 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:47.093+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:47.185 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:47.198+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:47.189 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 170.28368836199999, 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.190 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:24:47.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:24:47.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.86597824500001, 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:24:47.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.86179379999999, 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.202 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:24:47.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:24:47.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:24:47.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:24:47.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:24:47.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.77626271400001, 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.217 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:24:47.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:24:47.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:24:47.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.223 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.46228609900001, 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:24:47.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:24:47.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:24:47.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:24:47.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:24:47.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:24:47.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.460768255, 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:24:47.231 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.232 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:24:47.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:24:47.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.46010287999999, 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:24:47.240 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:24:47.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:24:47.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:24:47.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:24:47.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:24:47.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:24:47.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.45584270800001, 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:24:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:24:47.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:24:47.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.45520949900001, 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.258 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:24:47.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:24:47.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:24:47.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:24:47.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.454582397, 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:24:47.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:24:47.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:47.272 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.36632261899999, 2023-12-06T09:24:47.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:24:47.273 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.273 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:24:47.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 167.365401639, 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.279 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:24:47.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:24:47.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:24:47.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:24:47.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 166.61302370799999, 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:24:47.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:24:47.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.294 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.74298873699999, 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.741815226, 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:24:47.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:24:47.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:24:47.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:24:47.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.73366325500001, 2023-12-06T09:24:47.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:24:47.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:24:47.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:24:47.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:47.317 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.723381319, 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:47.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:24:47.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:24:47.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:24:47.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:24:47.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.72272343899999, 2023-12-06T09:24:47.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:24:47.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:24:47.337 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.338 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.338 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.339 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.339 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.72204866000001, 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:24:47.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:24:47.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:24:47.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:24:47.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:24:47.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 158.68969106899999, 2023-12-06T09:24:47.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:24:47.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:24:47.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.354 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.355 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.355 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.355 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.355 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:47.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:47.357 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:47.371+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:47.459 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:47.473+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:47.528 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:47.528 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:47.528 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:47.528 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:47.537 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:47.560 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:47.574+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:47.661 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:47.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:47.706 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:47.706 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:47.706 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:47.706 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:47.706 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:47.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:47.763 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:47.776+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:47.846 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 17.210501507, 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:24:47.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 16.905280206, 2023-12-06T09:24:47.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:24:47.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:47.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:24:47.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:24:47.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:47.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:47.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.861 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:47.862 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:47.862 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:47.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:47.864 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:47.877+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:47.940 INFO:teuthology.orchestra.run.smithi132.stdout:2.7 2023-12-06T09:24:47.964 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:47.978+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:48.034 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:24:48.034 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:48.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:48.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T09:24:48.088 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:48.102+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:48.170 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:48.185+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:48.172 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:48.172 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:24:48.172 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:24:48.172 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:24:48.172 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:48.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:48.272 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:48.286+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:48.352 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:24:48.352 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:24:48.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:24:48.352 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:24:48.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.352 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 153.39090877500001, 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:24:48.353 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:24:48.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:24:48.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:48.357 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:48.358 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:24:48.358 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:24:48.358 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:24:48.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:48.374 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:48.387+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:48.472 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 2.7 --file /var/log/ceph/exp.2.7.7 2023-12-06T09:24:48.473 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:48.488+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:48.491 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:48.492 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:48.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:48.574 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:48.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:48.670 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:48.670 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:48.670 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:48.675 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:48.689+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:48.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:48.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:24:48.804 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:48.818+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:48.863 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:48.863 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:48.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:48.863 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:24:48.863 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:48.869 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:48.876 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:48.890+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:48.977 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:48.991+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:49.003 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:49.004 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:49.004 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:24:49.004 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:49.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:49.030 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:24:49.031 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:24:49.114 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:24:49.114 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:24:49.115 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:49.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:49.142 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:24:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:24:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:24:49.143 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:49.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:49.197 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:49.211+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:49.201 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:24:49.201 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:24:49.285 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:49.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:49.288 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:24:49.288 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:24:49.326 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:24:49.326 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:24:49.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:24:49.326 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 170.61568071599999, 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.92122684700001, 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:24:49.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.47833437099999, 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:24:49.336 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:24:49.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.340 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.46950809000001, 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:24:49.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:24:49.350 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:24:49.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.46888641800001, 2023-12-06T09:24:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.46093993700001, 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:24:49.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:24:49.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:24:49.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:24:49.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:24:49.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.460465919, 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.368 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:24:49.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:24:49.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:24:49.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.46001156400001, 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.372 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:24:49.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:24:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:24:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:24:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 169.07601931299999, 2023-12-06T09:24:49.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:24:49.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 168.690869908, 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:24:49.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:24:49.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:24:49.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.387 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 168.678594597, 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 168.677109019, 2023-12-06T09:24:49.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:24:49.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:24:49.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:24:49.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:24:49.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.77057186100001, 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.765439558, 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:24:49.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:24:49.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.409 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.409 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:24:49.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.76461352800001, 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:24:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.76204406700001, 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:24:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.76129859700001, 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:24:49.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:24:49.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:24:49.433 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.76049252600001, 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:49.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:24:49.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:24:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.437 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 160.749388074, 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:24:49.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:24:49.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:24:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:24:49.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:49.444 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 11.976055937, 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:24:49.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:24:49.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:24:49.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:24:49.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:24:49.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:24:49.450 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.450 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.450 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.450 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:24:49.450 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:24:49.450 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:24:49.451 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:24:49.451 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:24:49.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:49.452 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:49.466+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:49.467 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:24:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:24:49.552 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:49.567+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:49.557 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:24:49.557 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:24:49.581 INFO:teuthology.orchestra.run.smithi132.stderr:Exporting 2.7 info 2.7( empty local-lis/les=44/45 n=0 ec=19/19 lis/c=44/44 les/c/f=45/45/0 sis=44) 2023-12-06T09:24:49.582 INFO:teuthology.orchestra.run.smithi132.stderr:Export successful 2023-12-06T09:24:49.630 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:49.630 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:49.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:49.654 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:49.668+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:49.757 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:49.769+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:49.767 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:49.767 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:49.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:49.856 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:49.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:49.905 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:49.905 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:49.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:49.959 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:49.973+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:50.045 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:50.045 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:50.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:50.058 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:50.072+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:50.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 2.7 2023-12-06T09:24:50.159 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:50.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:50.233 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:50.262 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:50.273+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:50.360 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:50.374+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:50.401 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:50.401 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:50.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:50.461 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:50.475+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:50.541 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:50.541 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:50.551 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:50.562 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:50.576+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:24:50.649 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T09:24:50.649 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T09:24:50.650 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T09:24:50.650 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:24:50.662 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:50.676+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:50.663 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:24:50.668+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 102755) UID: 0 2023-12-06T09:24:50.665 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:24:50.669+0000 9f54640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2023-12-06T09:24:50.736 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:50.736 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:50.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:50.763 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:50.777+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:50.864 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:50.878+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:50.937 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:50.937 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:50.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:50.988 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:51.002+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:51.068 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:51.083+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:51.123 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:51.123 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:51.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:51.170 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:51.184+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:51.207 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:41.223 102760== 2023-12-06T09:24:51.207 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:41.223 102760== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:24:51.219 INFO:tasks.ceph.mon.c.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:24:51.228 INFO:teuthology.orchestra.run.smithi132.stdout: marking collection for removal 2023-12-06T09:24:51.228 INFO:teuthology.orchestra.run.smithi132.stdout:setting '_remove' omap key 2023-12-06T09:24:51.228 INFO:teuthology.orchestra.run.smithi132.stdout:finish_remove_pgs 2.7_head removing 2.7 2023-12-06T09:24:51.228 INFO:teuthology.orchestra.run.smithi132.stdout:Remove successful 2023-12-06T09:24:51.271 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:51.285+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:51.313 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:51.313 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:51.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:51.374 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:51.388+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:51.475 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:51.486+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:51.501 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:51.502 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:51.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:51.597 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:51.611+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:51.676 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:51.690+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:51.681 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:51.681 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:51.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:51.745 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.7.7 2023-12-06T09:24:51.776 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:51.790+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:51.862 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:51.870 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:51.911 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:51.925+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:51.978 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:51.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:52.044 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:52.045 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:52.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:52.079 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:52.093+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:52.180 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:52.194+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:52.185 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:52.185 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:52.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:52.283 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:52.297+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:52.367 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:52.368 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:52.377 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:52.418 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:52.432+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:52.485 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:52.499+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:52.544 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:52.544 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:52.555 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:52.587 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:52.600+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:52.687 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:52.701+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:52.725 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:52.725 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:52.734 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:52.788 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:52.802+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:52.889 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:52.903+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:52.901 INFO:teuthology.orchestra.run.smithi132.stdout:get_pg_num_history pg_num_history pg_num_history(e51 pg_nums {1={15=1},2={19=8}} deleted_pools ) 2023-12-06T09:24:52.901 INFO:teuthology.orchestra.run.smithi132.stdout:Importing pgid 2.7 2023-12-06T09:24:52.903 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:52.903 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:52.907 INFO:teuthology.orchestra.run.smithi132.stdout:write_pg epoch 45 info 2.7( empty local-lis/les=44/45 n=0 ec=19/19 lis/c=44/44 les/c/f=45/45/0 sis=44) 2023-12-06T09:24:52.908 INFO:teuthology.orchestra.run.smithi132.stdout:Import successful 2023-12-06T09:24:52.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:53.000 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:53.014+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:53.084 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:53.084 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:53.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:53.094 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:53.107+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:53.192 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:53.206+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:53.263 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:53.263 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:53.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:53.293 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:53.307+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:53.394 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:53.408+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:53.400 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:53.400 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:53.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:53.427 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /var/log/ceph/exp.2.7.7 2023-12-06T09:24:53.496 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:53.510+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:53.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:53.597 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:53.610+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:53.691 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:53.691 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:53.697 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:53.711+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:53.702 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:53.798 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:53.812+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:53.870 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:53.871 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:53.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:53.922 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:53.936+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:54.000 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:54.014+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:54.049 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.049 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:54.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T09:24:54.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T09:24:54.102 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:54.116+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:54.186 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.186 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:54.203 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:54.217+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:54.304 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:54.318+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:54.327 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.327 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:54.405 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:54.419+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:54.466 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.466 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:54.505 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:54.519+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:54.607 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:54.621+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:54.648 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.648 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:54.708 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:54.722+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:54.785 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.785 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:54.839 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:54.853+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:54.909 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:54.923+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:54.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:55.018 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:55.032+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:55.101 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:55.101 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:55.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:55.112 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:55.125+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:55.212 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:55.226+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:55.281 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:55.282 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:55.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:55.313 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:55.327+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:55.414 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:55.428+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:55.421 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:55.421 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:55.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:55.515 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:55.529+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:55.570 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:55.570 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:55.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:55.625 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:55.639+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:55.709 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:55.710 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:55.717 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:55.731+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:55.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:55.818 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:55.832+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:55.886 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:55.886 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:55.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:55.919 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:55.933+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:56.020 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:56.034+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:56.065 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:56.066 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:56.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:56.121 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:56.135+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:56.222 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:56.236+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:56.244 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:56.244 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:56.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:56.323 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:56.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:56.387 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:56.387 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:56.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:56.439 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:56.453+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:56.524 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:56.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:56.568 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:56.568 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:56.579 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:24:56.625 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:56.640+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:56.710 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:56.710 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:56.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:24:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:24:56.752 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T09:24:56.752 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T09:24:56.753 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T09:24:56.753 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:24:56.761 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:56.776+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:56.768 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:24:56.771+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 104033) UID: 0 2023-12-06T09:24:56.769 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:24:56.773+0000 9f54640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2023-12-06T09:24:56.828 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:56.842+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:56.888 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:56.889 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:56.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:24:56.943 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:56.957+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:57.029 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:57.029 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:57.029 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:57.043+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:57.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:24:57.130 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:57.144+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:57.212 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:57.212 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:57.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:24:57.266 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:57.280+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:57.319 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:28.241 104038== 2023-12-06T09:24:57.320 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:28.241 104038== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:24:57.330 INFO:tasks.ceph.mon.a.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:24:57.331 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:57.346+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:57.354 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:57.355 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:57.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:24:57.450 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:57.464+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:57.533 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:57.533 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:57.534 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:57.549+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:57.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:24:57.635 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:57.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:57.714 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:57.715 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:57.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:24:57.737 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:57.752+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:57.838 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:57.852+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:57.858 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:57.858 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:57.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:24:57.951 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:57.965+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:58.035 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.035 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:58.040 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:58.054+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:58.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:24:58.141 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:58.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:58.213 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.213 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:58.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:24:58.242 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:58.257+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:58.343 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:58.358+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:58.388 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.388 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:58.400 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:24:58.444 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:58.459+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:58.496 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2023-12-06T09:24:58.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:24:58.497 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2023-12-06T09:24:58.497 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-06T09:24:58.497 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-06T09:24:58.499 INFO:tasks.ceph.osd.7:Started 2023-12-06T09:24:58.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:24:58.528 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.529 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:58.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:24:58.582 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:58.596+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:58.603 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T09:24:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:24:58.606 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:24:58.645 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:58.659+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:58.668 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.668 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:58.678 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:24:58.746 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:58.761+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:58.847 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.847 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:58.847 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:58.861+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:58.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:24:58.948 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:24:58.963+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:24:58.988 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:58.988 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:24:59.052 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:59.066+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:59.154 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:24:59.168+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:24:59.169 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:59.170 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.180 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:24:59.254 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:59.268+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:59.356 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:59.356 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.357 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:24:59.370+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:24:59.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:24:59.457 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:24:59.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:24:59.492 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:59.492 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:24:59.558 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:59.572+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:59.636 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:59.637 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:24:59.659 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:59.673+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:59.761 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:24:59.776+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:24:59.776 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:59.777 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:24:59.861 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:24:59.875+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:24:59.963 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:24:59.977+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:24:59.964 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:24:59.964 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:24:59.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:00.064 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:00.078+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:00.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-06T09:25:00.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T09:25:00.104 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:00.105 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:00.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:00.165 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:00.179+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:00.245 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:00.245 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:00.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:00.265 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:00.279+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:00.367 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:00.380+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:00.393 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:00.393 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:00.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:00.487 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:00.501+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:00.568 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:00.582+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:00.570 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:00.570 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:00.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:00.669 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:00.683+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:00.747 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:00.747 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:00.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:00.770 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:00.784+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:00.874 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:00.885+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:00.924 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:00.924 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:00.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:00.975 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:00.989+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:01.073 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:01.087+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:01.103 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:01.104 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:01.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:01.197 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:01.211+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:01.275 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:01.289+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:01.281 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:01.281 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:01.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:01.376 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:01.390+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:01.461 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:01.462 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:01.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:01.511 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:01.525+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:01.577 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:01.591+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:01.641 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:01.641 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:01.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:01.678 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:01.692+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:01.779 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:01.793+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:01.821 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:01.822 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:01.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:01.880 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:01.894+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:01.981 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:01.995+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:02.005 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:02.005 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:02.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:02.100 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:02.114+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:02.183 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:02.197+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:02.183 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:02.183 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:02.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:02.284 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:02.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:02.361 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:02.361 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:02.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:02.411 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:02.425+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:02.488 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:02.500+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:02.499 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:02.500 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:02.509 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:02.587 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:02.601+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:02.679 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:02.679 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:02.688 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:02.701+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:02.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:02.791 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:02.804+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:02.818 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:02.818 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:02.829 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:25:02.855 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T09:25:02.855 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T09:25:02.855 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2023-12-06T09:25:02.856 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T09:25:02.856 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T09:25:02.873 INFO:tasks.ceph.mon.b:Started 2023-12-06T09:25:02.873 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T09:25:02.873 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T09:25:02.876 INFO:tasks.ceph.mon.c:Started 2023-12-06T09:25:02.876 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:25:02.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:02.915 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:02.928+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:02.992 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:03.006+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:03.026 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:03.026 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:03.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:03.093 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:03.107+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:03.194 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:03.208+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:03.210 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:03.211 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:03.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:03.295 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:03.309+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:03.397 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:03.411+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:03.406 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:03.406 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:03.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:03.503 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:03.516+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:03.599 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:03.613+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:03.606 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:03.606 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:03.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:03.614 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:03.704 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:03.718+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:03.705 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T09:25:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:03.708 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:03.787 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:03.787 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:03.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:03.802 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:03.816+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:03.903 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:03.917+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:03.931 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:03.931 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:03.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:04.004 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:04.017+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:04.076 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:04.076 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:04.086 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:04.105 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:04.118+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:04.206 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:04.220+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:04.222 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:04.222 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:04.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:04.315 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:04.329+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:04.405 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:04.405 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:04.408 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:04.421+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:04.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:04.508 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:04.522+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:04.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:04.609 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:04.623+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:04.710 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:04.724+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:04.729 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:04.729 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:04.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:04.824 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:04.838+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:04.910 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:04.910 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:04.911 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:04.925+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:04.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:05.012 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:05.026+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:05.091 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:05.091 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:05.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:05.148 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:05.161+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:05.214 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:05.228+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:05.232 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:05.232 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:05.246 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:05.331 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:05.344+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:05.416 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:05.430+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:05.417 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:05.418 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:05.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:05.522 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:05.536+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:05.613 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:05.613 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:05.623 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:05.624 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:05.637+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:05.724 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:05.738+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:05.813 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:05.814 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:05.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:05.864 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:05.878+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:05.927 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:05.940+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:06.014 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:06.015 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:06.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:06.065 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:06.079+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:06.139 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:06.152+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:06.153 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:06.154 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:06.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:06.229 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:06.243+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:06.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-06T09:25:06.332 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:06.332 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:06.333 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:06.347+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:06.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:06.434 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:06.448+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:06.469 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:06.469 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:06.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:06.564 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:06.578+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:06.636 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:06.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:06.652 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:06.652 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:06.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:06.748 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:06.759+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:06.830 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:06.830 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:06.840 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:06.853+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:06.841 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:06.940 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:06.954+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:07.014 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.015 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:07.066 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:07.080+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:07.142 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:07.156+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:07.152 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.152 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.161 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:07.244 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:07.258+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:07.333 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.333 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:07.345 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:07.359+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:07.446 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:07.460+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:07.472 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.473 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:07.547 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:07.561+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:07.612 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.612 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:07.663 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:07.677+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:07.750 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:07.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:07.795 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.796 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.805 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:07.850 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:07.864+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:07.936 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:07.936 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:07.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:07.991 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:08.005+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:08.061 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:08.075+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:08.075 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:08.075 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:08.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:08.169 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:08.183+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:08.253 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:08.267+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:08.255 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:08.255 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:08.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:08.354 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:08.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:08.436 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:08.436 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:08.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:08.454 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:08.468+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:08.555 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:08.569+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:08.578 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:08.579 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:08.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:08.675 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:08.689+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:08.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:08.758 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:08.772+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:08.761 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:08.761 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:08.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:08.801 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T09:25:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:08.804 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:08.860 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:08.874+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:08.943 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:08.943 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:08.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:08.995 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:09.009+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:09.062 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:09.076+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:09.144 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:09.144 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:09.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:09.164 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:09.177+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:09.292 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:09.305+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:09.306 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:09.306 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:09.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:09.401 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:09.415+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:09.466 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:09.480+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:09.503 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:09.503 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:09.513 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:09.567 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:09.581+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:09.649 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:09.650 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:09.659 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:09.701 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:09.714+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:09.769 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:09.782+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:09.829 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:09.830 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:09.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:09.870 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:09.884+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:09.971 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:09.985+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:09.972 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:09.972 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:09.983 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:10.074 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:10.086+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:10.115 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:10.116 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:10.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:10.213 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:10.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:10.275 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:10.289+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:10.296 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:10.296 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:10.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:10.376 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:10.390+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:10.477 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:10.478 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:10.478 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:10.492+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:10.485 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:10.582 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:10.595+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:10.673 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:10.673 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:10.680 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:10.682 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:10.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:10.783 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:10.797+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:10.811 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:10.812 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:10.826 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:10.884 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:10.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:10.946 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:08.013 102582== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:10.955 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:10.955 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:10.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:10.985 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:10.999+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:10.992 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.054 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:11.086 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:11.101+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:11.106 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:11.106 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:11.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:11.182 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:08.248 102582== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:11.188 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:11.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:11.242 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.305 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:11.285 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:11.285 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:11.288 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:11.302+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:11.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:11.390 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:11.404+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:11.424 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:11.424 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:11.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:11.492 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:11.505+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:11.581 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:11.581 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:11.590 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:11.631 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:11.645+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:11.693 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:11.707+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:11.760 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:11.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:11.794 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:11.807+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:11.895 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:11.909+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:11.940 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:11.941 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:11.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:12.036 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:12.050+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:12.097 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:12.111+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:12.121 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:12.121 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:12.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:12.215 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:12.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:12.299 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:12.299 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:12.300 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:12.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:12.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:12.400 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:12.414+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:12.484 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:12.484 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:12.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:12.537 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:12.551+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:12.602 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:12.616+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:12.627 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:12.627 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:12.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:12.721 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:12.735+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:12.805 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:12.819+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:12.820 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:12.820 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:12.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:12.906 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:12.920+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:13.000 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.001 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.007 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:13.021+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:13.011 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:13.109 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:13.122+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:13.141 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.141 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:13.210 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:13.224+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:13.312 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:13.325+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:13.324 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.324 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.333 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:13.412 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:13.426+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:13.468 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.468 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:13.513 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:13.527+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:13.611 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.611 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.614 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:13.628+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:13.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:13.715 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:13.729+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:13.788 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.788 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:13.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:13.816 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:13.830+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:13.918 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:13.932+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:13.933 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:13.933 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:13.934 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T09:25:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:13.937 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:13.942 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:14.027 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:14.041+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:14.112 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:14.112 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:14.119 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:14.133+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:14.121 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:14.220 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:14.234+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:14.322 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:14.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:14.332 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:14.332 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:14.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:14.428 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:14.442+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:14.517 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:14.517 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:14.525 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:14.539+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:14.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:14.626 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:14.640+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:14.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:14.727 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:14.741+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:14.828 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:14.842+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:14.847 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:14.847 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:14.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:14.940 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:14.954+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:15.031 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:15.031 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:15.031 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:15.045+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:15.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:15.132 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:15.146+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:15.209 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:15.209 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:15.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:15.259 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:15.273+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:15.334 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:15.348+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:15.385 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:15.386 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:15.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:15.434 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:15.448+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:15.536 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:15.550+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:15.562 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:15.562 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:15.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:15.637 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:15.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:15.703 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:15.703 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:15.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:15.737 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:15.751+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:15.841 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:15.853+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:15.880 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:15.880 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:15.890 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:15.944 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:15.958+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:16.017 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.017 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:16.045 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:16.059+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:16.146 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:16.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:16.156 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.156 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:16.248 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:16.262+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.295 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.303 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:16.349 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:16.363+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:16.458 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:16.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:16.471 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.472 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:16.551 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:16.565+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:16.621 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.621 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:16.653 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:16.666+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:16.753 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:16.767+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:16.758 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.758 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:16.856 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:16.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:16.940 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:16.940 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:16.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:16.955 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:16.969+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:17.056 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:17.070+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:17.120 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:17.120 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:17.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:17.174 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:17.188+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:17.260 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:17.261 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:17.261 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:17.275+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:17.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:17.363 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:17.377+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:17.439 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:17.439 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:17.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:17.493 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:17.507+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:17.565 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:17.579+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:17.578 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:17.578 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:17.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:17.672 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:17.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:17.755 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:17.755 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:17.765 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:17.766 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:17.780+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:17.867 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:17.881+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:17.943 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:17.943 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:17.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:17.994 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:18.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:18.070 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:18.084+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:18.098 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:18.099 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:18.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:18.171 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:18.185+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:18.241 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:18.242 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:18.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:18.272 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:18.286+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:18.374 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:18.388+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:18.421 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:18.422 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:18.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:18.474 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:18.488+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:18.585 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:18.598+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:18.599 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:18.599 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:18.608 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:18.693 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:18.707+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:18.777 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:18.791+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:18.777 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:18.777 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:18.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:18.878 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:18.892+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:18.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:18.954 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:18.954 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:18.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:19.009 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:19.023+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T09:25:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:19.035 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:19.080 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:19.093+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:19.093 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:19.093 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:19.104 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:19.188 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:19.202+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:19.271 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:19.271 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:19.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:19.325 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:19.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:19.383 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:19.396+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:19.448 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:19.448 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:19.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:19.501 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:19.514+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:19.584 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:19.584 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:19.585 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:19.599+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:19.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:19.686 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:19.700+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:19.761 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:19.761 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:19.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:19.815 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:19.829+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:19.888 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:19.902+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:19.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:19.989 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:20.003+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:20.045 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.045 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:20.099 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:20.112+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:20.181 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.182 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:20.192 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:20.205+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:20.292 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:20.306+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:20.362 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.363 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:20.415 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:20.428+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:20.417 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:20.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:20.494 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:20.508+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:20.542 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.542 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:20.594 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:20.608+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:20.683 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.683 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:20.695 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:20.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:20.796 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:20.810+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:20.819 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.819 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.829 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:20.896 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:20.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:20.957 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:20.957 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:20.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:21.013 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:21.027+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:21.098 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:21.112+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:21.101 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:21.101 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:21.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:21.199 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:21.213+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:21.280 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:21.281 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:21.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:21.335 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:21.349+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:21.400 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:21.414+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:21.420 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:21.421 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:21.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:21.517 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:21.531+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:21.610 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:21.624+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:21.627 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:21.628 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:21.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:21.723 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:21.737+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:25:21.811 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:21.812 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:21.812 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:21.826+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:21.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:21.910 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:21.920+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:21.944 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:21.957 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:25:21.958 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T09:25:21.958 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T09:25:21.960 INFO:tasks.ceph.mon.a:Started 2023-12-06T09:25:21.960 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:25:21.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:21.999 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:21.999 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:22.007 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:22.021+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:22.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:22.108 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:22.122+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:22.184 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:22.184 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:22.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:22.212 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:22.226+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:22.313 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:22.327+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:22.331 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:22.331 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:22.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:22.364 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:23.794 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:22.414 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:22.428+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:22.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:25:22.430 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:23.859 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:22.515 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:22.529+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:22.521 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:22.521 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:22.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:22.616 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:22.630+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:22.703 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:22.704 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:22.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:22.718 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:22.719 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:22.720 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:22.733+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:22.750 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:22.812 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:22.812 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:22.818 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:22.832+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:22.913 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:22.913 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:22.922 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:22.936+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 52.284856767000001, 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:22.923 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:25:22.924 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:25:22.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:25:22.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 51.979635465999998, 2023-12-06T09:25:22.927 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.929 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:22.931 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:22.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:22.978 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:25:22.987+0000 5dc2340 -1 Falling back to public interface 2023-12-06T09:25:23.008 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:23.008 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:23.020 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:23.034+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:23.105 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:23.105 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:23.105 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:23.105 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:23.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:23.125 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:23.126 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:23.126 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:23.139+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:23.213 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:23.222 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:23.236+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:23.272 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:23.273 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:23.283 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:23.289 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:20.352 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:23.309 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:23.327 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:23.340+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:23.355 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:20.418 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:23.398 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:20.462 102582== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:23.425 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:24.854 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:23.427 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:23.441+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:23.445 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:23.445 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:23.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:23.445 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:23.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 188.48288991000001, 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:23.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:23.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:23.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:23.449 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:23.450 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:20.517 102582== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:23.455 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:23.527 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:23.541+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:23.562 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:20.624 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:23.589 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:23.589 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:23.589 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:23.590 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:23.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:23.631 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:23.645+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:23.645 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:20.712 102582== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:23.736 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:23.746+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:23.759 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:23.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:23.792 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:23.792 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:23.792 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:23.792 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:23.792 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:23.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:25:23.803 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:23.823 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:25.252 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:23.851 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:23.865+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:23.942 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:23.956+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:23.954 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:23.955 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:23.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:23.955 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:25:23.955 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:23.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:24.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:24.050 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:24.064+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:24.123 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:25.552 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:24.145 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:24.159+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:24.155 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:24.156 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:24.156 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:24.156 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:24.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:24.248 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2023-12-06T09:25:24.248 INFO:teuthology.orchestra.run.smithi132.stderr:0 2023-12-06T09:25:24.248 INFO:teuthology.orchestra.run.smithi132.stderr:1 2023-12-06T09:25:24.248 INFO:teuthology.orchestra.run.smithi132.stderr:2 2023-12-06T09:25:24.248 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2023-12-06T09:25:24.248 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2023-12-06T09:25:24.249 INFO:teuthology.orchestra.run.smithi132.stderr:config diff 2023-12-06T09:25:24.249 INFO:teuthology.orchestra.run.smithi132.stderr:config diff get 2023-12-06T09:25:24.249 INFO:teuthology.orchestra.run.smithi132.stderr:config get 2023-12-06T09:25:24.249 INFO:teuthology.orchestra.run.smithi132.stderr:config help [] 2023-12-06T09:25:24.249 INFO:teuthology.orchestra.run.smithi132.stderr:config set ... 2023-12-06T09:25:24.249 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2023-12-06T09:25:24.249 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:24.262+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:24.251 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:24.335 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:24.335 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:24.335 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:24.336 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:24.336 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:24.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:24.353 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:24.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:24.402 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:25.831 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:24.454 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:24.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:24.516 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:25:24.517 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.518 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 205.80649652299999, 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.519 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.520 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.521 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:25:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:25:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.522 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:25:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:25:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.523 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:25:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:25:24.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 205.11204265399999, 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.525 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.526 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.527 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:25:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:25:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.528 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.529 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.530 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.669150178, 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.531 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.532 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:24.539 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:25:24.540 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:25:24.541 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.542 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.66032389700001, 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.543 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.544 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:25:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:25:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.545 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:25:24.546 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:25:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:25:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.547 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.548 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:24.549 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.65970222499999, 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:25:24.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:25:24.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.553 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.65175574400001, 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.554 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.555 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.556 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:25:24.557 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:25:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:25:24.558 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:25:24.559 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.65128172600001, 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.560 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.561 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.562 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.563 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.65082737099999, 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.564 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.565 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:25:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:25:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:25:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:25:24.569 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 204.26683512, 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.570 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.571 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.572 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:25:24.573 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.881685715, 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.574 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.575 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:25:24.576 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:25:24.579 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.579 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.579 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.579 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:25:24.579 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:25:24.580 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.86941040400001, 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.581 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.582 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.583 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.584 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.86792482600001, 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.586 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:25:24.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:25:24.591 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.96138766799999, 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.593 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.594 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.595 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.596 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.956255365, 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:24.597 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.599 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.599 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.599 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:24.599 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.599 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:25:24.600 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:25:24.601 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.602 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:25:24.605 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.95542933499999, 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.606 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:25:24.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.609 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.610 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:25:24.611 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.95285987400001, 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.612 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.613 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.614 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.615 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:25:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:25:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.629 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:25:24.630 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.95211440400001, 2023-12-06T09:25:24.632 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.633 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.634 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.635 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.636 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.95130833299999, 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:24.637 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.638 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.639 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.640 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:25:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:25:24.641 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.642 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 195.940203881, 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:25:24.643 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:24.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:25:24.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 47.166871743999998, 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:25:24.648 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:25:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:24.649 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:24.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:25:24.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:25:24.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:25:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:25:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:25:24.663 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:24.664 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:24.664 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:24.666 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:24.679+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:24.667 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:24.668 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:26.096 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:24.771 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:24.785+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:24.839 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:24.839 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:24.839 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:24.839 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:24.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:24.891 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:24.904+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:24.942 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:26.371 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:24.973 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:24.987+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:25.038 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:25.038 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:25.038 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:25.038 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:25.038 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:25.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:25.074 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:25.088+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:25.107 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:25:25.174 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:25.175 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:25.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:25.175 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:25:25.175 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:25.175 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:25.189+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:25.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:25.217 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:26.647 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:25.276 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:25.290+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:25.315 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:25.316 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:25.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:25.379 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:25.391+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:25.454 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:25.454 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:25.454 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:25.455 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:25.455 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:25.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:25.481 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:26.911 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:25.511 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:25.525+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:25.579 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:25.592+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:25.595 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:25.595 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:25.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:25.595 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:25:25.595 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:25.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:25.690 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:25.704+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:25.750 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:27.179 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:25.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:25:25.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:25.782 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:25.795+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:25.793 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:25.793 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:25.793 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:25.793 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:25.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:25.891 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:25.905+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:25.980 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:25.980 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:25.980 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:25.981 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:25.981 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:25.984 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:25.998+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:25.989 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:26.086 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:26.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:26.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-12-06T09:25:26.101+0000 7f5252ffd640 0 --2- 172.21.15.88:0/4178193603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f525413c990 0x7f525413cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-06T09:25:26.187 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:26.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:26.217 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:26.217 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.308183848, 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:25:26.218 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:26.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:26.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:25:26.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:25:26.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:25:26.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:25:26.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.89047373099999, 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:25:26.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:25:26.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:25:26.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:25:26.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:25:26.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:25:26.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:25:26.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:25:26.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:25:26.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:25:26.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:25:26.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:25:26.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.248 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.88628928599999, 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:26.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:25:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.80075819999999, 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.257 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:25:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:25:26.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.264 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.48678158499999, 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:26.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:25:26.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:25:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:25:26.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:25:26.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:25:26.270 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.270 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.270 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.48526374100001, 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.272 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:26.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:26.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.279 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.484598366, 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.280 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.285 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.48033819400001, 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.286 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:25:26.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.47970498500001, 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:25:26.294 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:25:26.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:25:26.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:25:26.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.299 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.479077883, 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.300 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.39081810499999, 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:25:26.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:25:26.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:25:26.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.310 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 206.389897125, 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.311 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 205.63751919399999, 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.317 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:25:26.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:25:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:25:26.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.767484223, 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.322 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.327 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.76631071200001, 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.332 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.75815874099999, 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:25:26.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.747876805, 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.338 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.747218925, 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:25:26.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.74654414599999, 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.349 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:25:26.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:25:26.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:25:26.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:25:26.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:25:26.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:25:26.357 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.357 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.358 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.358 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:26.358 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:26.358 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 197.714186555, 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:25:26.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:26.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:25:26.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:26.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:25:26.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:25:26.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:25:26.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:25:26.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.368 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:26.369 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:26.369 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:26.370 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:27.704 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:26.370 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:27.735 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:26.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:26.372 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:26.384+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:26.402 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:26.403 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:26.511 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:26.532 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:26.533 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:26.533 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:26.547+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:26.570 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:26.571 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:26.571 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:26.571 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:26.572 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:26.586+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:26.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:26.637 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:26.637 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:26.673 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:26.687+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:26.776 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:26.776 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:26.777 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:26.790+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:26.787 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:26.842 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:26.843 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:26.877 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:26.891+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:26.928 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:26.929 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:26.959 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:26.959 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:26.972 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:26.978 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:26.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:27.030 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:27.033 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:27.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:27.079 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:27.093+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:27.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:25:27.147 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:27.147 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:27.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:27.180 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:27.194+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:27.281 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:27.295+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:27.336 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:27.336 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:27.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:27.391 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:27.405+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:27.475 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:27.476 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:27.482 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:27.496+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:27.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:27.583 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:27.597+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:27.624 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:27.624 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:27.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:27.718 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:27.732+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:27.785 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:27.799+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:27.807 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:27.807 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:27.817 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:27.887 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:27.901+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:27.988 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:28.002+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:27.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:28.089 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:28.103+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:28.164 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:28.165 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:28.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:28.190 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:28.204+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:28.291 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:28.305+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:28.307 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:28.307 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:28.315 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:28.392 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:28.406+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:25:28.467 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:28.467 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:28.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:28.493 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:28.507+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:28.594 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:28.608+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:28.615 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:28.615 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:28.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:28.695 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:28.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:28.793 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:28.793 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:28.796 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:28.810+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:28.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:28.897 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:28.911+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:28.939 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:28.939 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:28.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:28.998 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:29.012+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:29.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:25:29.078 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:29.078 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:29.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:29.104 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:29.115+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:29.203 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:29.216+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:29.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:29.331 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:29.332 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:29.333 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:29.346+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:29.334 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:29.334 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:29.334 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:29.334 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:29.334 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:29.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:29.431 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:29.445+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:29.460 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:29.461 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:29.504 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:29.518+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:29.515 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:25:29.516 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:29.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:29.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:29.565 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:29.566 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:29.609 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:29.623+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:29.662 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:29.707 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:29.721+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:29.719 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:29.719 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:29.719 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:29.719 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:29.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:29.780 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:29.780 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:29.808 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:29.822+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:29.867 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:29.867 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:29.922 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:29.922 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:29.922 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:29.923 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:29.923 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:29.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:29.959 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:29.971+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:29.959 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:29.962 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:29.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:30.011 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:30.024+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:30.111 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:30.125+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:30.156 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 213.248936988, 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.157 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:30.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:25:30.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.164 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.83122687100001, 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.167 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.82704242599999, 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.74151133999999, 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.179 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.42753472499999, 2023-12-06T09:25:30.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.185 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.42601688100001, 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.190 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:25:30.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:25:30.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:25:30.197 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.425351506, 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.198 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:25:30.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.42109133400001, 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.42045812500001, 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.209 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:30.214 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.419831023, 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:25:30.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.33157124499999, 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.220 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 210.330650265, 2023-12-06T09:25:30.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.57827233399999, 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.231 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:25:30.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.236 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.70823736299999, 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:25:30.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.707063852, 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.242 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.69891188099999, 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:25:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:25:30.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:25:30.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:25:30.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.252 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.688629945, 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:25:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:25:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.257 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.687972065, 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.258 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.68729728599999, 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.654939695, 2023-12-06T09:25:30.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:25:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:25:30.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.273 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:30.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:30.276 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:30.276 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.225 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:30.277 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:30.289+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:30.376 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:30.390+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:30.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:25:30.452 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:30.452 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:30.452 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:30.452 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:30.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:30.477 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:30.491+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:30.494 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:08.472 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:30.579 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:30.593+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:30.638 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:30.639 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:30.639 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:30.639 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:30.639 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:30.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:30.680 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:30.694+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:30.779 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:30.779 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.143789341000002, 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:25:30.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:25:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:30.783 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.838568039999998, 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:25:30.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:25:30.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:30.787 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:30.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:30.789 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:30.803+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:30.904 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:30.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:30.919 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:30.919 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:30.919 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:30.919 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:30.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:30.992 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:31.006+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:31.093 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:31.107+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:31.118 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:31.118 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:31.119 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:31.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:31.129 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2023-12-06T09:25:31.129 INFO:teuthology.orchestra.run.smithi132.stderr:0 2023-12-06T09:25:31.129 INFO:teuthology.orchestra.run.smithi132.stderr:1 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:2 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs debug_inject_read_zeros 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs files list 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs stats 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore bluefs device info [] 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:config diff 2023-12-06T09:25:31.130 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2023-12-06T09:25:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:31.131 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:31.194 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:31.208+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:31.295 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:31.309+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 196.337623974, 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:31.299 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:31.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:25:31.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:31.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:31.303 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:31.308 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:31.381 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:32.810 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:31.397 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:31.411+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:31.439 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:31.440 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:31.440 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:31.440 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:31.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:31.498 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:31.511+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:25:31.594 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:31.594 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:31.594 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:31.603 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:31.617+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:31.605 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:31.700 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:31.714+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:31.773 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:31.773 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:31.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:31.773 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:25:31.773 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:31.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:31.825 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:31.838+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:31.903 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:31.916+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:31.953 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:31.953 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:31.954 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:31.954 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:31.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:31.966 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:25:32.011 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:32.025+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:32.104 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:32.118+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:32.143 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:32.143 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:32.143 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:32.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:32.204 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:32.205 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:32.206 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:32.218+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:32.297 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:32.319 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:25:32.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 213.61459863799999, 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:25:32.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.92014476899999, 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:32.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:25:32.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:32.327 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.47725229299999, 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.46842601200001, 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:25:32.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:25:32.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.335 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.46780433999999, 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.339 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.45985785900001, 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.459383841, 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:25:32.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.45892948599999, 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:32.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:25:32.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.352 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 212.07493723499999, 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:32.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:25:32.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:25:32.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:25:32.356 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 211.68978783, 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:25:32.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:32.360 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 211.677512519, 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 211.676026941, 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:25:32.364 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:25:32.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:25:32.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:25:32.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.76948978300001, 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.368 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.76435748, 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.76353144999999, 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.376 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:25:32.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:25:32.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:25:32.379 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.76096198900001, 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:25:32.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:25:32.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.383 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.76021651900001, 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.75941044800001, 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 203.748305996, 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:32.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:25:32.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:25:32.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.395 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 54.974973859000002, 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:32.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:32.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:32.399 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:32.400 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:32.339+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:32.400 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:32.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:32.407 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:32.421+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:32.480 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:32.480 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:32.508 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:32.522+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:32.571 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:32.571 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:32.571 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:32.571 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:32.585 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:32.586 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:32.587 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:32.630 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:32.644+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:32.693 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:32.710 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:32.723+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:32.733 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:32.733 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:32.733 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:32.733 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:32.733 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:32.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:32.790 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:32.794 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:32.799 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:32.840 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:32.854+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:32.911 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:32.911 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:32.912 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:32.925+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:32.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:33.013 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:33.027+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:33.050 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:33.051 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:33.060 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:33.145 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:33.159+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:33.215 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:33.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:33.232 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:33.232 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:33.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:33.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:33.316 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:33.330+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:33.417 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:33.431+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:33.422 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:33.422 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:33.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:33.518 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:33.532+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:33.563 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:33.563 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:33.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:33.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:25:33.658 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:33.672+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:33.734 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:33.748+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:33.752 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:11.730 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:33.755 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:33.755 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:33.765 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:33.823 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:11.801 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:33.823 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:33.837+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:33.921 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:33.935+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:33.960 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:33.960 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:33.974 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:33.979 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:34.034 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:34.048+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:34.102 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:12.080 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:34.122 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:34.136+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:34.141 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:34.142 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:34.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:34.237 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:34.251+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:34.324 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:34.338+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:34.331 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:34.331 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:34.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:34.428 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:34.442+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:34.518 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:34.518 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:34.528 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:34.529 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:34.542+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:34.629 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:34.643+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:34.701 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:34.702 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:34.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:34.730 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:34.744+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:34.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:25:34.832 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:34.846+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:34.887 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:34.888 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:34.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:34.934 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:34.947+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:35.035 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:35.049+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:35.055 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:35.055 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:35.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:35.136 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:35.150+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:35.201 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:35.202 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:35.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:35.238 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:35.251+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:35.245 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:35.338 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:35.352+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:35.341 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:35.341 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:35.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:35.439 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:35.453+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:35.522 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:35.522 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:35.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:35.539 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:35.553+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:35.562 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:35.641 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:35.655+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:35.653 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:35.658 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:35.658 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:35.667 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:35.744 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:35.756+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:35.761 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:35.762 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:35.803 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:35.804 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:35.814 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:35.859 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:35.872+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:35.860 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:35.946 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:25:35.982 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:35.983 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:35.983 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:35.996+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:36.025 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:36.025 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:36.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:36.025 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:25:36.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.025 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 217.31851172399999, 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:25:36.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:25:36.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.62405785499999, 2023-12-06T09:25:36.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.18116537899999, 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:36.040 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.17233909800001, 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:25:36.044 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.17171742599999, 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:25:36.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:25:36.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.049 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.16377094500001, 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.163296927, 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 216.16284257199999, 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:25:36.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:25:36.062 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:36.063 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 215.77885032099999, 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:25:36.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:25:36.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:25:36.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:25:36.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 215.393700916, 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.069 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:25:36.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:25:36.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 215.381425605, 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:25:36.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:25:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:25:36.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 215.379940027, 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:25:36.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:36.083 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.47340286900001, 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:25:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:36.088 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.468270566, 2023-12-06T09:25:36.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:25:36.088 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.088 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.088 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:25:36.092 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.46744453599999, 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:25:36.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.46487507500001, 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.104 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.46412960500001, 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.46332353400001, 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 207.452219082, 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:36.113 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 58.678886945000002, 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:25:36.118 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:36.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:36.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:25:36.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:36.122 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:36.123 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:36.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:36.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:36.126 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:36.138+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:36.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:36.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:36.225 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:36.239+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:36.297 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:36.297 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:36.298 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:36.298 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:36.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:36.351 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:36.366+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:36.427 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:36.442+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:36.438 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:36.439 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:36.439 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:36.439 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:36.439 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:36.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:36.492 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:0 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:1 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:2 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator dump block 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator fragmentation block 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator fragmentation histogram block [] [] 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator score block 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:config diff 2023-12-06T09:25:36.493 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2023-12-06T09:25:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:36.495 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:36.536 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:36.551+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:36.616 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:36.616 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:36.616 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:36.616 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:25:36.616 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:36.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:36.666 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:36.681+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:36.734 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:38.162 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:36.736 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:38.166 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:36.739 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:36.752+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:36.753 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:36.754 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:36.754 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:36.754 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:36.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:36.847 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:36.861+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:36.934 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:36.949+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:36.940 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:36.940 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:36.940 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:36.940 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:36.940 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:36.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:36.995 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:37.035 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:37.050+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:37.123 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:37.123 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:37.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:37.123 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:25:37.123 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:37.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:37.138 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:37.153+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:37.239 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:37.253+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:37.261 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:37.261 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:37.261 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:37.262 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:37.271 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:37.345 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:37.360+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:37.443 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:37.443 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:37.443 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:37.443 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:37.443 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:37.446 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:37.461+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:37.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:37.570 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:37.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:37.641 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 220.733324404, 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:25:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 218.31561428699999, 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:25:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:25:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:25:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:25:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 218.311429842, 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:25:37.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:25:37.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:25:37.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.664 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 218.22589875599999, 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:37.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:25:37.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.91192214099999, 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.671 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.676 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.91040429700001, 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:25:37.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.909738922, 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.683 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:25:37.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:25:37.689 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.90547874999999, 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.690 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:37.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:25:37.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:25:37.695 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.90484554099999, 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:25:37.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:25:37.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.904218439, 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.702 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:37.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:25:37.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.815958661, 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.708 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:37.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:25:37.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:25:37.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.81503768100001, 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.715 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:25:37.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:25:37.720 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 217.06265974999999, 2023-12-06T09:25:37.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:25:37.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.192624779, 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:25:37.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:25:37.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.19145126800001, 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:25:37.734 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:25:37.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:25:37.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.18329929699999, 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.740 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.745 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.17301736100001, 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.746 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:25:37.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:25:37.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:25:37.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:25:37.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.172359481, 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.751 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:25:37.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.17168470199999, 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.757 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:25:37.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 209.139327111, 2023-12-06T09:25:37.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:37.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:25:37.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:25:37.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:25:37.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:25:37.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:25:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:37.771 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:37.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:37.774 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:37.774 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:25:37.774 INFO:tasks.mon_recovery:removing mon b 2023-12-06T09:25:37.775 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T09:25:37.775 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:25:37.775 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:37.787+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:37.789 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:25:37.794+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 102578) UID: 0 2023-12-06T09:25:37.790 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:25:37.795+0000 9f54640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-06T09:25:37.874 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:37.888+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:37.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:25:37.969 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:37.969 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:37.969 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:37.969 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:37.980 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:37.993+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:37.981 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:38.076 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:38.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:38.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:25:38.179 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:38.193+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:38.206 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:38.206 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:38.206 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:38.206 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:38.207 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:38.217 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:38.273 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:35.339 102582== 2023-12-06T09:25:38.273 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:35.339 102582== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:25:38.280 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:38.295+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:38.283 INFO:tasks.ceph.mon.b.smithi132.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:25:38.383 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:38.397+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:38.391 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:38.391 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:38.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:38.391 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 67.755341895000001, 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:38.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.395 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 67.450120593999998, 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:38.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:25:38.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:25:38.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:38.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:25:38.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:25:38.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.402 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:38.403 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.403 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.403 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:38.403 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:38.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:38.486 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:38.499+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:38.554 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:38.555 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:38.555 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:38.555 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:38.566 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:38.607 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:38.621+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:38.685 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:38.699+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:38.740 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:38.740 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:38.752 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:38.753 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:38.753 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:38.753 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:38.753 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:38.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:38.805 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:38.818+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:38.837 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:38.837 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:38.924 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:38.924 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:38.924 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:38.937+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:38.953 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:38.953 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 203.992082501, 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:38.954 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:38.955 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:25:38.956 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.957 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:38.958 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.959 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:38.959 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:38.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:39.008 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:39.018+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:39.011 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:39.012 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:39.091 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:39.106+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:39.101 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:39.102 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:39.138 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:39.138 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:39.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:39.192 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:39.192 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:39.206+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:39.277 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:39.277 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:39.279 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:39.280 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:39.286 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:39.287 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:39.291 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:39.331 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:39.344+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:39.394 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:39.408+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:39.415 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:39.415 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:39.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:39.495 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:39.510+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:39.596 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:39.610+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:39.596 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:39.597 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:39.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:39.699 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:39.713+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:39.780 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:39.780 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:39.790 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:39.799 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:39.813+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:39.904 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:39.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:39.918 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:39.919 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:39.928 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:40.001 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:40.016+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:40.097 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:40.097 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:40.104 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:40.118+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:40.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:40.204 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:40.219+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:40.238 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:40.238 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:40.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:40.307 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:40.320+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:40.376 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:40.376 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:40.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:40.395 INFO:tasks.ceph.osd.7.smithi132.stderr:==00:00:00:41.823 102302== Warning: unimplemented fcntl command: 1038 2023-12-06T09:25:40.407 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:40.421+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:40.511 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:40.523+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:40.517 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:40.517 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:40.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:40.611 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:40.625+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:40.654 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:40.654 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:40.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:40.712 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:40.726+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:40.813 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:40.827+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:40.831 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:40.832 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:40.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:40.926 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:40.940+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:41.011 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:41.012 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:41.016 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:41.029+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:41.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:41.116 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:41.130+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:41.194 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:41.194 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:41.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:41.218 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:41.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:41.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:25:41.333 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:41.333 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:41.333 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:41.345+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:41.341 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:41.430 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:41.439+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:41.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:41.513 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:41.513 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:41.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:41.567 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:41.580+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:41.629 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:41.643+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:41.650 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:41.650 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:41.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:41.749 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:41.761+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:41.831 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:41.844+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:41.835 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:41.835 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:41.843 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:41.933 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:41.946+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:42.011 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.011 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.025 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:42.034 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:42.048+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:42.136 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:42.149+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:42.154 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.154 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:42.237 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:42.250+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:42.294 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.294 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:42.348 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:42.362+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:42.439 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:42.452+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:42.440 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.440 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:42.540 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:42.553+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:42.620 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.621 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:42.641 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:42.654+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:42.745 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:42.758+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:42.759 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.759 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:42.842 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:42.856+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:42.944 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:42.944 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:42.945 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:42.958+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:42.949 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:43.045 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:43.059+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:43.079 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:43.079 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:43.088 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:43.146 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:43.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:43.247 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:43.261+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:0 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:1 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:2 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs debug_inject_read_zeros 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs files list 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs stats 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator dump block 2023-12-06T09:25:43.263 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator fragmentation block 2023-12-06T09:25:43.264 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2023-12-06T09:25:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:43.266 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:43.266 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:43.266 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:43.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:43.350 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:43.364+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:43.449 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:43.449 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:43.449 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:43.463+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:43.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:43.553 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:43.566+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:43.587 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:43.587 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:43.596 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:43.653 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:43.667+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:43.726 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:43.726 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:43.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:43.754 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:43.768+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:43.855 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:43.869+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:25:43.876 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T09:25:43.877 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T09:25:43.877 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:25:43.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:43.881 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:43.881 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:43.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:43.956 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:43.970+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:44.061 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:44.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:44.066 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:44.066 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:44.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:44.158 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:44.172+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:44.205 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:44.206 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:44.215 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:44.258 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:44.272+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:44.359 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:44.373+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:44.384 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:44.384 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:44.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:44.460 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:44.474+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:44.561 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:44.574+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:44.567 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:44.567 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:44.568 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:44.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:44.661 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:44.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:44.705 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:44.706 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:44.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:44.762 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:44.776+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:44.863 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:44.877+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:44.882 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:44.883 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:44.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:44.964 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:44.978+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:45.060 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.061 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.065 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:45.079+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:45.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:45.183 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:45.197+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:45.198 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.199 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:45.268 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:45.281+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:45.369 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:45.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:45.374 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.374 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:45.470 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:45.484+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:45.512 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.512 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:45.570 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:45.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:45.671 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:45.685+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:45.690 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.690 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.699 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:45.772 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:45.786+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:45.829 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.829 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:45.873 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:45.887+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:45.972 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:45.972 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:45.974 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:45.988+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:45.978 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:46.075 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:46.089+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:46.163 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:46.164 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:46.174 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:46.176 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:46.190+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:46.277 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:46.291+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:46.306 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:46.306 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:46.314 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:46.378 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:46.392+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:46.479 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:46.493+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:46.484 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:46.484 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:46.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:46.580 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:46.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:46.664 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:46.664 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:46.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:46.680 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:46.694+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:46.781 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:46.795+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:46.804 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:46.804 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:46.814 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:46.882 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:46.896+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:46.983 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:46.997+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:46.984 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:46.985 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:46.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:47.085 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:47.099+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:47.126 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:47.127 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:47.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:47.186 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:47.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:47.286 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:47.286 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:47.287 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:47.300+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:47.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:47.387 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:47.401+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:47.472 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:47.472 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:47.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:47.522 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:47.536+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:47.529 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:25:47.539+0000 5dc2340 -1 osd.7 51 log_to_monitors true 2023-12-06T09:25:47.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:25:47.590 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:47.603+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:47.683 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:47.683 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:47.691 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:47.705+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:47.692 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:47.792 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:47.806+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:47.821 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:47.821 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:47.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:47.916 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:47.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:47.994 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:48.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:48.002 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.002 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:48.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:48.095 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:48.109+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:48.176 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.177 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:48.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:48.229 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:48.243+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:48.267 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:48.299 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:48.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:48.314 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.314 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:48.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:48.399 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:48.413+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:48.418 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2023-12-06T09:25:48.418 INFO:teuthology.orchestra.run.smithi132.stderr:0 2023-12-06T09:25:48.418 INFO:teuthology.orchestra.run.smithi132.stderr:1 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:2 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs debug_inject_read_zeros 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs files list 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs stats 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator dump block 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator fragmentation block 2023-12-06T09:25:48.419 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2023-12-06T09:25:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:48.421 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:48.491 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.491 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:48.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:48.501 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:48.514+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:48.614 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:48.628+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:48.630 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.630 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:48.642 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:48.726 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:48.740+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:48.804 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:48.818+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:48.811 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.812 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:48.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:48.905 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:48.919+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:48.991 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:48.991 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.000 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:49.044 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:49.058+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:49.107 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:49.121+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:49.129 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:49.129 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.139 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:49.208 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:49.222+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:49.268 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:49.268 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.278 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:49.322 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:49.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:49.410 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:49.424+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:49.421 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:49.421 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:49.511 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:49.525+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:49.598 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:49.599 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.608 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:49.612 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:49.626+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:49.713 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:49.727+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:49.737 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:49.737 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:49.814 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:49.827+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:49.873 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:49.873 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:49.882 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:49.924 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:49.938+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:50.015 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:50.029+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:50.051 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:50.051 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:50.061 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:50.116 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:50.130+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:50.192 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:50.192 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:50.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:50.218 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:50.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:50.319 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:50.332+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:50.369 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:50.369 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:50.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:50.419 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:50.433+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:50.521 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:50.535+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:50.547 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:50.547 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:50.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:50.622 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:50.636+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:50.689 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:50.689 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:50.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:50.741 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:50.755+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:50.824 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:50.838+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:50.872 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:50.872 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:50.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:50.965 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:50.979+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:51.027 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:51.041+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:51.050 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:51.050 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:51.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:51.143 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:51.157+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:51.231 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:51.245+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:51.239 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:51.240 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:51.250 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:51.334 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:51.348+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:51.417 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:51.417 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:51.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:51.467 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:51.481+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:51.534 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:51.548+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:51.593 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:51.594 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:51.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:51.634 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:51.648+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:51.734 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:51.734 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:51.735 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:51.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:51.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:51.836 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:51.850+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:51.880 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:51.880 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:51.889 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:51.973 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:51.987+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:52.038 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:52.052+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:52.058 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:52.058 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:52.066 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:52.139 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:52.153+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:52.233 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:52.233 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:52.240 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:52.254+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:52.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:52.341 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:52.354+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:52.369 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:52.369 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:52.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:52.441 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:52.455+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:52.543 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:52.556+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:52.545 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:52.545 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:52.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:52.643 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:52.657+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:52.697 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:52.697 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:52.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:52.752 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:52.766+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:52.833 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:52.834 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:52.842 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:52.845 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:52.859+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:52.947 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:52.960+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:52.959 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:52.959 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:53.011 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:53.011 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:53.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:53.048 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:53.048 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:53.062 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:53.076+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:53.132 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:53.132 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:53.148 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:53.162+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:53.189 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:53.189 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:53.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:53.215 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:53.283 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:53.297+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:53.302 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:53.303 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:53.353 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:53.366+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:53.411 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:53.412 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:53.412 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:53.412 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:53.412 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:53.412 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:53.422 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:53.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:53.463 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:53.477+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:53.500 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:53.500 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:53.511 INFO:teuthology.orchestra.run.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-12-06T09:25:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T09:25:53.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T09:25:53.557 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:53.571+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:53.597 INFO:teuthology.orchestra.run.smithi132.stderr:no valid command found; 10 closest matches: 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:0 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:1 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:2 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:abort 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:assert 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs debug_inject_read_zeros 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs files list 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:bluefs stats 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator dump block 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:bluestore allocator fragmentation block 2023-12-06T09:25:53.598 INFO:teuthology.orchestra.run.smithi132.stderr:admin_socket: invalid command 2023-12-06T09:25:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T09:25:53.601 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2023-12-06T09:25:53.650 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:25:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.650 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 234.944681616, 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:25:53.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:25:53.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:25:53.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 234.250227747, 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:53.659 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.807335271, 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:25:53.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:25:53.663 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.79850898999999, 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:25:53.664 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.667 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.79788731799999, 2023-12-06T09:25:53.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:25:53.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:25:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:25:53.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.78994083699999, 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:53.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:25:53.678 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.78946681900001, 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.681 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:25:53.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:25:53.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.789012464, 2023-12-06T09:25:53.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:25:53.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:25:53.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.405020213, 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.690 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.691 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.692 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:25:53.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.01987080800001, 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.695 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:25:53.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.00759549700001, 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 233.00610991900001, 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:25:53.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.09957276099999, 2023-12-06T09:25:53.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:25:53.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:25:53.713 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.09444045800001, 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:25:53.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:25:53.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.093614428, 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:25:53.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.09104496699999, 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:25:53.723 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:25:53.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:25:53.727 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.09029949699999, 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:25:53.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.08949342599999, 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:25:53.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 225.07838897400001, 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:25:53.737 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:25:53.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.741 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 76.305056836999995, 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:25:53.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:25:53.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:25:53.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:25:53.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.746 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.746 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:25:53.746 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:25:53.746 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:53.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:53.750 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:53.761+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:53.848 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:53.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:53.880 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:53.880 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:53.880 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:25:53.880 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:53.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:25:53.982 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:53.996+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:54.018 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:54.018 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:25:54.018 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:25:54.018 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:25:54.018 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:54.027 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:54.050 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:54.064+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:54.151 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:54.165+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:54.153 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:25:54.153 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:25:54.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:25:54.153 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:25:54.153 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:25:54.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:54.252 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:54.266+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:54.295 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:54.295 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:54.295 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:54.295 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:54.305 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:54.353 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:54.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:54.435 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:54.436 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:54.436 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:54.436 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:54.436 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:54.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:54.461 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:25:54.464 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:54.475+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:54.476 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:25:54.476 INFO:tasks.mon_recovery:causing some monitor log activity 2023-12-06T09:25:54.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2023-12-06T09:25:54.556 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:54.570+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:54.592 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:54.592 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:54.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:54.592 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:25:54.592 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:54.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:54.657 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:54.671+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:54.741 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:54.742 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:54.742 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:54.742 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:54.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:54.758 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:54.772+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:54.760 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:25:54.762+0000 10253640 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T09:25:54.859 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:54.873+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:54.928 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:54.928 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:54.928 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:54.928 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:54.928 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:54.938 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:54.959 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:54.973+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:55.060 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:55.074+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:55.161 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:55.175+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:55.197 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:55.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 238.28452199099999, 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.198 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:25:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:25:55.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:55.204 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.86681187400001, 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:25:55.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.86262742900001, 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.210 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:25:55.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:25:55.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:25:55.215 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.77709634300001, 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:25:55.216 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:25:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:25:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.46311972800001, 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:25:55.225 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:25:55.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:25:55.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:25:55.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:25:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:25:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.461601884, 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.234 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:25:55.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.46093650899999, 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.241 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.242 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:25:55.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.242 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:25:55.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.456676337, 2023-12-06T09:25:55.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:25:55.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.253 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.456043128, 2023-12-06T09:25:55.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:25:55.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.45541602599999, 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.266 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.36715624799999, 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:25:55.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:25:55.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:25:55.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:25:55.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 235.366235268, 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.272 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:25:55.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.278 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 234.61385733700001, 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:25:55.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:25:55.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.74382236599999, 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:25:55.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:25:55.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:25:55.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:25:55.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.742648855, 2023-12-06T09:25:55.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.73449688400001, 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.296 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:25:55.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.301 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.724214948, 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:25:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.72355706799999, 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:25:55.307 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:25:55.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.72288228900001, 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.314 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:25:55.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:25:55.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:25:55.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:25:55.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.320 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.69052469799999, 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:25:55.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:55.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:55.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:55.327 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:55.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:55.329 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:55.343+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:55.429 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:55.443+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:55.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:25:55.522 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:55.522 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:55.523 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:55.523 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:55.533 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:55.544+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:55.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:55.631 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:55.645+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:55.726 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:55.726 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:55.726 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:55.726 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:55.726 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:55.732 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:55.746+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:55.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:55.778 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:55.875 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:55.875 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:55.876 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:55.877 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:55.891+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:55.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:25:55.960 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:55.960 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:55.978 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:55.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:56.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2023-12-06T09:25:56.053 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.053 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.066 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:56.069 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:56.078 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:56.092+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:56.167 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:56.179 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:56.193+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:56.226 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.226 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:25:56.251 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:56.251 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:56.285 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:56.294+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:56.351 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:56.352 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:56.377 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.378 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.381 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:56.395+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:56.387 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:25:56.483 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:56.496+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:56.501 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:25:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:25:56.560 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.560 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:25:56.583 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:56.597+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:56.684 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:56.698+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:56.705 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.705 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.715 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:25:56.785 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:56.799+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:56.850 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.851 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:25:56.904 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:56.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:56.987 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:56.987 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:56.988 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:57.001+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:56.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:25:57.088 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:57.102+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:57.165 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:57.165 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:57.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:25:57.189 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:25:57.203+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:25:57.290 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:57.304+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:57.303 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:57.303 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:57.313 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:25:57.399 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:57.411+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:57.482 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:57.482 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:57.492 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:25:57.493 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:57.506+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:57.500 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:25:57.594 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:57.607+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:57.661 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:57.662 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:57.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:25:57.712 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:57.726+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:57.795 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:57.809+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:57.838 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:57.838 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:57.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:25:57.896 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:57.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:57.976 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:57.976 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:57.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:25:58.034 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:58.045+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:58.099 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:58.111+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:58.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2023-12-06T09:25:58.125 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:58.125 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:58.134 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:25:58.198 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:58.212+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:58.299 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:58.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:58.307 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:58.307 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:58.317 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:25:58.399 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:58.413+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:58.452 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:58.452 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:58.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:25:58.500 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:58.514+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:58.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:25:58.599 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:55.659 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.599 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:55.661 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.601 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:58.615+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:58.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:25:58.641 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:55.703 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.653 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:58.653 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:58.655 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:36.632 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.656 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:36.634 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:25:58.705 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:25:58.719+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:25:58.738 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:58.739 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:58.746 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:36.724 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.805 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:58.819+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:58.848 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:25:58.848 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:25:58.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:25:58.919 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:25:58.927+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:25:58.938 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:56.000 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.939 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:56.002 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.941 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:36.920 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.942 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:36.921 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.953 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:36.931 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.953 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:56.016 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:25:58.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:25:58.992 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:25:59.008 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:59.022+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:59.109 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:25:59.123+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:25:59.135 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:25:59.135 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:25:59.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 242.284673766, 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:25:59.211 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:25:59.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.86696364900001, 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:25:59.218 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:25:59.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:25:59.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:25:59.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.223 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.86277920399999, 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.224 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:25:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:25:59.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.77724811799999, 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:25:59.230 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.46327150299999, 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.236 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.241 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.46175365900001, 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.461088284, 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.247 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:25:59.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.45682811200001, 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.45619490300001, 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.258 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:25:59.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:25:59.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.455567801, 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:25:59.263 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:25:59.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:25:59.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:25:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:25:59.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.36730802299999, 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.269 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:25:59.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:25:59.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:25:59.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:25:59.276 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.366387043, 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.277 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:25:59.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:25:59.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:25:59.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 238.61400911199999, 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:25:59.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:25:59.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.743974141, 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.289 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:25:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:25:59.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:25:59.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.74280063, 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:25:59.294 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:25:59.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:25:59.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.73464865899999, 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:59.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:25:59.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:25:59.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:25:59.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.724366723, 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.305 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.310 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.723708843, 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:25:59.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:25:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.72303406399999, 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:25:59.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:25:59.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:25:59.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 230.690676473, 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:25:59.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.327 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:59.328 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:25:59.224+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:25:59.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:25:59.329 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:25:59.329 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:25:59.331 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:59.345+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:59.344 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:25:59.345 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:25:59.431 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:25:59.445+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:25:59.434 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:25:59.434 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:25:59.478 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:59.478 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:59.478 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:59.485 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:25:59.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:25:59.518 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:25:59.518 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:25:59.566 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:59.579+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:59.627 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:25:59.627 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:25:59.635 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:25:59.649+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:25:59.641 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:59.641 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:59.641 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:25:59.641 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:25:59.642 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:59.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:25:59.735 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:25:59.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:25:59.738 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:25:59.738 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:25:59.835 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:59.835 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 89.199961353000006, 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:25:59.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:25:59.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 88.894740052000003, 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:59.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:25:59.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:25:59.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:25:59.843 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:25:59.844 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:59.844 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:25:59.850+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:25:59.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:25:59.937 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:25:59.950+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:25:59.986 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:25:59.986 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:25:59.986 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:25:59.986 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:25:59.998 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:00.044 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:00.058+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:00.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2023-12-06T09:26:00.144 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:00.144 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:00.144 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:00.145 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:00.145 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:00.145 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:00.158+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:00.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:00.240 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:00.254+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:00.339 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:00.339 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:00.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:00.339 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:00.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.339 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 225.37731808199999, 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:00.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:26:00.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:26:00.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:00.343 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:00.344 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:00.359+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:00.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:00.465 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:00.479+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:00.479 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:00.479 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:00.479 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:00.479 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:00.488 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:00.574 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:00.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:00.649 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:00.662+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:00.658 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:00.658 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:00.658 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:00.658 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:00.658 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:00.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:00.750 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:00.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:00.836 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:00.836 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:00.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:00.838 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:00.838 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:00.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:00.851 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:00.864+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:00.952 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:00.965+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:00.974 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:00.975 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:00.975 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:00.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:01.052 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:01.066+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:01.117 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:01.117 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:01.117 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:01.117 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:01.117 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:01.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:26:01.187 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:01.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:01.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2023-12-06T09:26:01.263 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:01.277+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:01.338 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:01.338 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:01.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 242.632327412, 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:01.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.342 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.93787354299999, 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:01.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:01.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.494981067, 2023-12-06T09:26:01.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.353 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:01.354 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:01.354 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.48615478599999, 2023-12-06T09:26:01.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:01.355 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.356 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.356 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.359 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.48553311399999, 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.47758663299999, 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:01.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:01.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:01.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:01.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.47711261500001, 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.374 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.375 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:01.376 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.377 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:01.378 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:01.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.380 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.47665825999999, 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:01.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:01.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 241.092666009, 2023-12-06T09:26:01.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:01.386 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.386 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.386 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:01.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 240.70751660400001, 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:01.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:01.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:01.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:01.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 240.69524129300001, 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:01.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.402 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 240.69375571500001, 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.78721855699999, 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.407 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:01.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.415 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.78208625400001, 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.419 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.78126022399999, 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.424 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.77869076299999, 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:01.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:01.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.77794529299999, 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:01.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:01.434 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.77713922199999, 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:01.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 232.76603477, 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 83.992702632999993, 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:01.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:01.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.447 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:01.448 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:01.448 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:01.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:01.450 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:01.463+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:01.551 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:01.564+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:01.577 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:01.577 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:01.577 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:01.586 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:01.651 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:01.665+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:01.752 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:01.765+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:01.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:01.773 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:01.773 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:01.774 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:01.774 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:01.774 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:01.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:01.853 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:01.866+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:01.953 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:01.953 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:01.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:01.953 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:01.954 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:01.954 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:01.967+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:01.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:02.016 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:02.016 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:02.054 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:02.068+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:02.100 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:02.133 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:02.134 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:02.134 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:02.134 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:02.146 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:02.155 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:02.169+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:02.185 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:02.185 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:02.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2023-12-06T09:26:02.257 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:02.270+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:02.278 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:02.279 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:02.317 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:02.318 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:02.318 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:02.318 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:02.318 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:02.328 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:02.369 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:02.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:02.378 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:02.459 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:02.473+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:02.465 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:02.466 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:02.511 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:02.511 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:02.511 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:02.511 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:02.511 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:02.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:02.562 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:02.562 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:02.562 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:02.576+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:02.646 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:02.646 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:02.655 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:02.655 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:02.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:02.663 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:02.677+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:02.762 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:02.777+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:02.834 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:02.834 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:02.843 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:02.888 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:02.901+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:02.965 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:02.979+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:02.972 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:02.972 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:02.981 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:03.066 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:03.080+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:03.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:26:03.150 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:03.150 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:03.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:03.199 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:03.213+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:03.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2023-12-06T09:26:03.299 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:03.312+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:03.334 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:03.334 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:03.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:03.387 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:03.401+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:03.470 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:03.484+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:03.517 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:03.517 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:03.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:03.570 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:03.585+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:03.674 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:03.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:03.727 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:03.727 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:03.733 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:03.748 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:03.748 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:03.748 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2023-12-06T09:26:03.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2023-12-06T09:26:03.777 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:03.791+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:03.874 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:03.888+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:03.885 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:03.885 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:03.894 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:03.979 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:03.993+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:04.072 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:04.073 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:04.076 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:04.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:04.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:04.178 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:04.192+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:04.254 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:04.254 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:04.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:04.307 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:04.321+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:04.379 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:04.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:04.395 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:04.395 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:04.405 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:04.490 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:04.504+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:moved 0 / 18 (0%) 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:avg 2.25 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:stddev 1.19896 -> 1.19896 (expected baseline 1.40312) 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:min osd.1 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:max osd.3 with 5 -> 5 pgs (2.22222 -> 2.22222 * mean) 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:oload 110 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:max_change 3 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:max_change_osds 4 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:average_utilization 0.0003 2023-12-06T09:26:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:overload_utilization 0.0003 2023-12-06T09:26:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:no change 2023-12-06T09:26:04.543 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2023-12-06T09:26:04.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2023-12-06T09:26:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:04.588 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:04.588 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:04.601+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:04.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:04.682 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:04.695+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:04.695 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:26:04.729 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:04.729 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:04.739 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:04.782 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:04.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:04.884 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:04.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:04.906 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:04.906 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:04.912 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:04.912 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:04.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:04.986 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:05.000+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:05.023 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:05.023 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:05.053 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:05.053 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:05.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:05.140 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:05.140 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:05.140 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:05.153+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:05.186 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:05.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:05.192 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:05.193 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:05.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout:moved 0 / 18 (0%) 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout:avg 2.25 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout:stddev 1.19896 -> 1.19896 (expected baseline 1.40312) 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout:min osd.1 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout:max osd.3 with 5 -> 5 pgs (2.22222 -> 2.22222 * mean) 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:26:05.207 INFO:teuthology.orchestra.run.smithi088.stdout:oload 1000 2023-12-06T09:26:05.208 INFO:teuthology.orchestra.run.smithi088.stdout:max_change 1 2023-12-06T09:26:05.208 INFO:teuthology.orchestra.run.smithi088.stdout:max_change_osds 4 2023-12-06T09:26:05.208 INFO:teuthology.orchestra.run.smithi088.stdout:average_utilization 0.0003 2023-12-06T09:26:05.208 INFO:teuthology.orchestra.run.smithi088.stdout:overload_utilization 0.0029 2023-12-06T09:26:05.208 INFO:teuthology.orchestra.run.smithi088.stderr:no change 2023-12-06T09:26:05.223 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2023-12-06T09:26:05.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2023-12-06T09:26:05.231 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:05.231 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:05.287 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:05.301+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:05.322 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:05.322 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:05.391 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:05.402+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:05.416 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:05.416 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:05.418 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:05.418 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:05.419 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:05.419 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:05.419 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:05.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:05.489 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:05.503+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:05.525 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:05.526 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:05.591 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:05.604+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:05.646 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 246.919928309, 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.651 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:05.652 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:05.653 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:05.654 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 246.22547444, 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.655 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:05.656 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:05.657 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.658 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.782581964, 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.659 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.660 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.661 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.662 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:05.662 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:05.662 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.662 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.663 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.663 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:05.663 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:05.663 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.663 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.666 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:05.666 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:05.666 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.77375568299999, 2023-12-06T09:26:05.668 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:05.668 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.668 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.668 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:05.670 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:05.671 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.773134011, 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.672 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.673 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.674 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.675 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.676 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.677 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.76518752999999, 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.678 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:05.680 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.681 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.682 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.683 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.76471351199999, 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:05.684 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:05.685 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:05.686 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:05.687 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:05.688 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.689 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.764259157, 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.690 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:05.692 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.692 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.692 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:05.693 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.694 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.695 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:05.696 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.380266906, 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.697 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.698 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.699 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:05.700 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:05.701 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 244.99511750100001, 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.702 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:05.703 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.704 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:05.705 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 244.98284219000001, 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.706 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:05.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:05.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.710 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 244.98135661200001, 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.711 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.712 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.713 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.714 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.07481945399999, 2023-12-06T09:26:05.715 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.716 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.717 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:05.718 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.719 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.06968715100001, 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.720 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.721 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.722 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.723 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.068861121, 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.724 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:05.725 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:05.726 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.06629165999999, 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:05.728 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.729 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.730 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.06554618999999, 2023-12-06T09:26:05.732 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.733 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.734 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.735 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.06474011899999, 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.737 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.738 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:05.739 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.740 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 237.05363566700001, 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.741 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.742 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.743 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.744 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 88.280303529999998, 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:05.745 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:05.746 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:05.746 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:05.746 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.746 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:05.747 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:05.747 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:05.748 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:05.749 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:05.750 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:05.751 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:05.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:05.753 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:05.766+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:05.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2023-12-06T09:26:05.849 INFO:teuthology.orchestra.run.smithi088.stdout:moved 0 / 18 (0%) 2023-12-06T09:26:05.849 INFO:teuthology.orchestra.run.smithi088.stdout:avg 2.25 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:stddev 1.19896 -> 1.19896 (expected baseline 1.40312) 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:min osd.1 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:max osd.3 with 5 -> 5 pgs (2.22222 -> 2.22222 * mean) 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:oload 1000 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:max_change 1 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:max_change_osds 4 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:average_utilization 0.0003 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stdout:overload_utilization 0.0029 2023-12-06T09:26:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:no change 2023-12-06T09:26:05.854 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:05.868+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:05.866 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2023-12-06T09:26:05.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2023-12-06T09:26:05.920 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:05.920 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:05.921 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:05.921 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:05.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:05.974 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:05.988+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:06.055 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:06.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:06.059 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:06.060 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:06.060 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:06.060 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:06.060 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:06.070 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:06.157 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:06.171+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:06.242 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:06.242 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:06.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:06.243 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:06.243 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:06.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:06.258 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:06.271+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:06.359 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:06.372+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:06.382 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:06.383 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:06.383 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:06.383 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:06.392 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:06.462 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:06.473+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:moved 0 / 18 (0%) 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:avg 2.25 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:stddev 1.19896 -> 1.19896 (expected baseline 1.40312) 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:min osd.1 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:max osd.3 with 5 -> 5 pgs (2.22222 -> 2.22222 * mean) 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:oload 110 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:max_change 1 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:max_change_osds 4 2023-12-06T09:26:06.511 INFO:teuthology.orchestra.run.smithi088.stdout:average_utilization 0.0003 2023-12-06T09:26:06.512 INFO:teuthology.orchestra.run.smithi088.stdout:overload_utilization 0.0003 2023-12-06T09:26:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:no change 2023-12-06T09:26:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:06.523 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2023-12-06T09:26:06.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2023-12-06T09:26:06.528 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:06.565 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:06.579+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:06.654 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:06.655 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:06.655 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:06.655 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:06.655 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:06.663 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:06.676+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:06.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:06.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2023-12-06T09:26:06.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:26:06.763 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:06.778+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:06.842 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:06.843 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:06.843 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:06.843 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:06.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:06.899 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:06.913+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:06.972 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:06.986+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:06.999 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:06.999 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:06.999 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:06.999 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:06.999 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:07.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:07.075 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:07.089+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:07.176 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:07.190+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:07.190 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:07.190 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:07.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:07.191 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:26:07.191 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:07.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:07.218 INFO:teuthology.orchestra.run.smithi088.stdout:moved 0 / 18 (0%) 2023-12-06T09:26:07.218 INFO:teuthology.orchestra.run.smithi088.stdout:avg 2.25 2023-12-06T09:26:07.218 INFO:teuthology.orchestra.run.smithi088.stdout:stddev 1.19896 -> 1.19896 (expected baseline 1.40312) 2023-12-06T09:26:07.218 INFO:teuthology.orchestra.run.smithi088.stdout:min osd.1 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T09:26:07.218 INFO:teuthology.orchestra.run.smithi088.stdout:max osd.3 with 5 -> 5 pgs (2.22222 -> 2.22222 * mean) 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stdout:oload 110 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stdout:max_change 0.05 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stdout:max_change_osds 4 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stdout:average_utilization 0.0003 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stdout:overload_utilization 0.0003 2023-12-06T09:26:07.219 INFO:teuthology.orchestra.run.smithi088.stderr:no change 2023-12-06T09:26:07.281 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:07.294+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:07.330 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:07.330 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:07.330 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:07.331 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:07.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:07.382 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:07.395+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:07.482 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:07.492+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:07.514 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:07.515 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:07.515 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:07.515 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:07.515 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:07.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:07.583 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:07.595+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:07.682 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:07.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:07.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2023-12-06T09:26:07.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 250.82102395999999, 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:26:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:26:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:26:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:26:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:26:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:26:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:26:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 248.40331384300001, 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:26:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:26:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 248.39912939800001, 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:26:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:26:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 248.31359831200001, 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:26:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:26:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:26:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.99962169700001, 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:26:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:26:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.998103853, 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.99743847799999, 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:26:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:26:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:26:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:26:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.993178306, 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.992545097, 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:26:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:26:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:26:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.99191799499999, 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:26:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:26:07.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:26:07.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:26:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.833 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.90365821699999, 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:07.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:26:07.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:26:07.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:26:07.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.902737237, 2023-12-06T09:26:07.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:26:07.840 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.840 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.840 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.840 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.846 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 247.15035930600001, 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.847 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:26:07.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.850 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:26:07.852 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.28032433499999, 2023-12-06T09:26:07.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:26:07.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:26:07.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.279150824, 2023-12-06T09:26:07.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:26:07.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:26:07.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:26:07.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.27099885300001, 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.866 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.869 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.871 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.260716917, 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:26:07.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:07.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.26005903699999, 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.878 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:26:07.883 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.884 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.884 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.884 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.25938425800001, 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:07.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:26:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:26:07.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 239.22702666699999, 2023-12-06T09:26:07.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:26:07.893 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:07.894 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:07.894 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:07.894 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:07.894 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:26:07.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:07.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:26:07.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:26:07.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:07.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:07.904 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:07.905 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:07.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:07.909 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:26:07.910 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:07.920+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:08.007 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:08.021+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:08.011 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:08.011 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:08.080 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:08.080 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:08.081 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:08.081 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:08.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:08.100 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:08.100 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:08.111 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:08.125+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:08.187 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:08.187 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:08.212 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:08.226+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:08.246 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:08.247 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:08.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:08.276 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:08.313 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:08.327+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:08.365 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:08.365 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:08.407 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:08.407 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:08.414 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:08.428+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:08.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:08.458 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:08.515 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:08.529+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:08.541 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:08.541 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:08.586 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:08.586 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:08.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:08.624 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:08.625 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:08.639+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:08.701 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2023-12-06T09:26:08.735 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:08.735 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:08.736 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:08.750+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:08.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:08.833 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:08.847+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:08.923 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:08.923 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:08.924 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:08.938+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:08.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:09.025 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:09.039+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:09.065 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.066 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:09.127 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:09.141+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:09.206 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.206 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:09.228 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:09.241+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:09.329 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:09.342+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:09.342 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.342 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:09.431 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:09.445+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:09.480 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.480 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:09.534 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:09.548+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:09.619 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.619 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.629 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:09.636 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:09.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:09.739 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:09.753+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:09.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2023-12-06T09:26:09.758 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.759 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.768 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:09.852 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:09.866+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:09.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:26:09.942 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:09.943 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:09.943 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:09.957+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:09.947 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:10.044 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:10.058+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:10.116 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:10.116 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:10.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:10.146 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:10.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:10.249 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:10.263+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:10.257 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:10.257 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:10.266 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:10.354 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:10.365+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:10.395 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:10.395 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:10.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:10.490 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:10.504+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:10.553 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:10.567+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:10.575 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:10.576 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:10.585 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:10.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:10.687 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:10.701+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:10.752 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:10.752 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:10.756 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:10.770+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:10.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:10.857 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:10.871+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:10.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2023-12-06T09:26:10.898 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:26:10.900 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:10.901 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:10.946 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:10.946 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:10.946 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:10.946 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:10.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:10.958 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:10.972+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:10.999 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:11.000 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:11.059 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:11.073+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:11.109 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:11.109 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:11.139 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:11.139 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:11.139 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:11.139 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:11.140 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:11.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:11.159 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:11.174+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:11.197 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:11.197 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:11.261 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:11.274+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:11.281 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:11.281 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:11.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:11.281 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:26:11.281 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:11.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:11.298 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:11.299 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:11.362 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:11.375+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:11.384 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:11.385 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:11.451 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:11.452 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:11.452 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:11.452 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:11.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:11.468 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:11.468 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:11.468 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:11.482+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:11.554 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:11.554 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:11.562 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:11.577+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:11.618 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:11.618 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:11.618 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:11.618 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:11.618 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:11.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:11.664 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:11.678+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:11.765 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:11.779+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:26:11.814 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 254.907587057, 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:26:11.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:26:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:26:11.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.48987693999999, 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.821 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:11.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:26:11.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:26:11.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:26:11.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:26:11.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:26:11.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.485692495, 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.828 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:26:11.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:26:11.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:26:11.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.842 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.40016140899999, 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.843 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:11.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:26:11.845 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:26:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.847 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:26:11.848 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.08618479399999, 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:26:11.849 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.850 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.850 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.851 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:26:11.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.853 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.854 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:26:11.855 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:11.856 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.08466695000001, 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.857 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:26:11.858 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:26:11.859 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:26:11.860 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.861 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.084001575, 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.862 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.863 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:26:11.864 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.865 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.866 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.867 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.868 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.07974140300001, 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:26:11.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.870 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.871 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:11.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.871 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.871 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.871 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:26:11.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:26:11.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:26:11.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:26:11.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:26:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:26:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:26:11.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.07910819400001, 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:26:11.878 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.884 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.078481092, 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 251.990221314, 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.891 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:26:11.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:26:11.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.897 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 251.98930033400001, 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.898 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:26:11.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:26:11.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:26:11.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:26:11.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 251.23692240299999, 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:26:11.904 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:26:11.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:26:11.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:26:11.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:26:11.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.366887432, 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.910 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:11.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:26:11.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:26:11.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:26:11.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.917 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.36571392100001, 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.918 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:26:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:26:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:26:11.921 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:26:11.922 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.35756194999999, 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:26:11.923 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:11.924 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:26:11.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:26:11.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.34728001400001, 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.929 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.930 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:26:11.932 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.933 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.933 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.933 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.934 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.935 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.346622134, 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.936 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.937 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:26:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.34594735499999, 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.942 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:26:11.943 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:26:11.944 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:26:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:26:11.946 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 243.313589764, 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:11.947 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.948 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.949 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:11.950 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:26:11.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.952 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.953 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.953 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:11.953 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:11.953 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:11.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:11.955 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2023-12-06T09:26:11.956 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:11.968+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:12.055 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:12.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:12.133 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:12.134 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:12.134 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:12.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:12.157 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:12.170+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:12.239 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:12.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:12.239 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-12-06T09:26:12.239 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2023-12-06T09:26:12.239 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2023-12-06T09:26:12.284 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:12.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:12.314 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:12.314 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:12.314 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:12.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:12.365 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:12.379+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:12.460 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:12.474+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:12.538 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:12.538 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 101.902349036, 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:12.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:12.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:12.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:12.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 101.597127735, 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:12.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:12.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:12.547 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:12.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:12.590 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:12.603+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:12.662 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:12.676+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:12.732 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:12.732 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:12.732 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:12.733 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:12.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:12.764 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:12.778+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:12.866 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:12.880+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:12.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:26:12.923 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:12.923 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:12.924 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:12.924 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:12.924 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:12.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:12.966 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:12.980+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:12.972 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2023-12-06T09:26:13.068 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:13.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:13.158 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 238.19627554799999, 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:13.159 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:13.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:13.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:13.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:13.163 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:13.169 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:13.169 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:13.183+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:13.270 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:13.283+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:13.308 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:13.309 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:13.309 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:13.309 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:13.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:13.402 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:13.416+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:13.471 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:13.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:13.484 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:13.485 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:13.485 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:13.485 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:13.485 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:13.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:13.566 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:13.579 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:13.593+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:13.664 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:13.664 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:13.664 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:13.664 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:13.664 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:13.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:13.674 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:13.687+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:13.807 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:13.808 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:13.808 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:13.821+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:13.842 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:13.842 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:13.842 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:13.842 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:13.851 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:13.894 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:13.908+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:13.977 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:13.990+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:13.978 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:13.978 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:13.978 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:13.978 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:13.978 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:13.982 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:13.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:14.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2023-12-06T09:26:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.2 primary-affinity to 1 (8655362) 2023-12-06T09:26:14.084 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:14.098+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:14.089 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:26:14.177 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:14.178 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:14.192+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:14.201 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:14.201 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 255.49452760599999, 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.202 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:14.203 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:14.204 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.205 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.206 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:14.207 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:14.208 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.80007373699999, 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:14.209 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.212 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.212 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:14.213 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:14.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.213 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.214 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.35718126099999, 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.34835498000001, 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:14.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.222 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.34773330799999, 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.224 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.227 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:14.227 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:14.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:14.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:14.229 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.229 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.229 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:14.230 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:14.230 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.230 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.231 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.33978682700001, 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:14.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.235 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.33931280900001, 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:14.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:14.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:14.239 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 254.33885845399999, 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:14.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:14.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:14.243 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 253.95486620299999, 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.248 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:14.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:14.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:14.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:14.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 253.569716798, 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.254 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:14.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.257 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 253.55744148700001, 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.258 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:14.259 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:14.260 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:14.261 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 253.55595590900001, 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:14.262 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:14.263 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:14.264 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:14.265 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.266 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.64941875100001, 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.267 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.268 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.269 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.270 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.644286448, 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.271 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.272 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.273 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.274 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.64346041799999, 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.275 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:14.276 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:14.277 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:14.278 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.64089095700001, 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.279 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:14.280 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:14.281 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:14.282 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.64014548700001, 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:14.283 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.284 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.285 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:14.286 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.287 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.63933941600001, 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:14.291 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 245.628234964, 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:14.292 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:14.293 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:14.294 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:14.295 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 96.854902827000004, 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:14.296 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.298 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:14.300 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:14.301 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:14.301 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:14.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:14.312 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:14.325+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:14.371 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:14.371 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:14.402 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:14.416+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:14.459 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:14.459 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:14.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:14.528 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:14.542+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:14.604 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:14.618+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:14.634 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:14.634 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:14.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:14.730 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:14.743+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:14.808 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:14.820+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:14.812 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:14.812 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:14.821 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:14.909 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:14.923+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:14.992 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:14.992 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:15.043 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:15.057+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:15.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2023-12-06T09:26:15.111 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:15.124+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:15.171 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:15.172 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:15.211 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:15.225+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:15.309 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:15.309 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.312 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:15.326+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:15.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:15.413 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:15.427+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:15.465 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:15.465 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.475 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:15.519 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:15.533+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:15.605 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:15.606 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.615 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:15.615 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:15.628+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:15.716 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:15.730+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:15.798 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:15.798 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:15.852 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:15.866+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:15.918 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:15.932+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:15.935 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:15.936 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:15.945 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:16.019 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:16.034+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:16.118 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:16.118 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:16.121 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:16.135+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:16.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:16.168 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:26:16.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2023-12-06T09:26:16.222 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:16.237+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:16.275 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:16.275 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:16.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:16.330 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:16.343+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:16.424 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:16.439+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:16.441 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:16.442 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:16.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:16.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:16.535 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:16.549+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:16.625 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:16.626 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:16.626 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:16.640+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:16.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:16.728 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:16.742+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:16.807 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:16.807 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:16.816 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:16.822 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:16.822 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:16.828 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:16.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:16.912 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:16.912 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:16.939 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:16.953+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:16.977 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:16.977 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:16.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:17.005 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:17.005 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:17.049 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:17.063+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:17.095 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:17.096 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:17.131 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:17.145+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:17.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2023-12-06T09:26:17.148 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:17.148 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:17.148 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:17.148 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:17.148 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:17.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:17.193 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:17.232 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:17.246+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:17.275 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:26:17.285 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:17.285 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:17.332 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:17.346+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:17.345 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:17.346 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:17.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:17.346 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:17.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.346 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:17.346 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 242.38384625200001, 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:26:17.347 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:17.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.348 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:17.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:17.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:17.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:17.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:17.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:17.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:26:17.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:26:17.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:17.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:26:17.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:26:17.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:17.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:26:17.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:17.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:17.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:17.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:26:17.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:26:17.354 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:17.354 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:17.355 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:17.355 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:17.355 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:17.355 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:17.357 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:17.373 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:17.373 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:17.433 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:17.447+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:17.464 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:17.465 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:17.496 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:17.496 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:17.496 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:17.496 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:17.505 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:17.551 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:17.566+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:17.633 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:17.633 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:17.633 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:17.633 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:17.634 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:17.636 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:17.650+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:17.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:17.737 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:17.751+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:17.814 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:17.814 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:17.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:17.814 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:17.814 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:17.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:17.865 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:17.879+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:17.939 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:17.953+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:17.990 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:17.990 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:17.990 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:17.990 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:17.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:18.040 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:18.054+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:18.125 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:18.126 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:18.126 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:18.134 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:18.141 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:18.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:18.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2023-12-06T09:26:18.242 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:18.257+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:18.302 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 259.59691724499999, 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.303 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:18.304 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:18.305 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:18.306 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.90246337600001, 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.308 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:18.309 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.310 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.311 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.45957090000002, 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.312 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.313 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.313 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:18.314 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:18.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.316 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.45074461899998, 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.317 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.318 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:18.319 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.320 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:18.321 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.45012294700001, 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.322 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:18.323 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.324 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:18.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.326 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.44217646599998, 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:18.327 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.328 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:18.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:18.330 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:18.331 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.441702448, 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.332 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.333 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.334 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:18.334 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:18.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.334 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.334 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.335 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:18.336 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:18.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.336 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.336 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.337 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.44124809300001, 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.338 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.339 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.340 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.341 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 258.05725584200002, 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.342 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.343 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.346 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 257.67210643700002, 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.347 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.348 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:18.349 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.350 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 257.65983112599997, 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.351 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:18.352 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:18.353 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.354 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 257.658345548, 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.357 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.358 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.75180839000001, 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.359 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.360 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:18.360 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.361 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.361 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:18.362 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:18.363 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.363 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.363 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.363 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:18.363 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:18.364 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.364 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.364 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.364 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:18.365 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.366 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.366 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:18.366 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:18.366 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.366 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.367 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.367 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:18.367 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:18.367 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:18.368 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.746676087, 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.369 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:18.370 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:18.371 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:18.372 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.373 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.374 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.375 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:18.376 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:18.376 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.74585005700001, 2023-12-06T09:26:18.376 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:18.376 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.376 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.376 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.377 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.377 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.379 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:18.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.74328059600001, 2023-12-06T09:26:18.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:18.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:18.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.384 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:18.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.387 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.74253512600001, 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:18.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:18.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.74172905500001, 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:18.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:18.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 249.730624603, 2023-12-06T09:26:18.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:18.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:18.402 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 100.957292466, 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:18.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:18.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:18.406 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:18.407 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:18.411 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:18.411 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:18.423+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:18.512 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:18.526+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:18.578 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:18.579 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:18.579 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:18.579 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:18.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:18.638 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:18.653+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:18.661 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:15.724 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:18.663 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:15.726 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:18.685 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:15.747 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:18.713 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:56.691 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:18.713 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:18.727+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:18.713 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:56.692 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:18.724 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:18.724 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:18.724 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:18.724 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:18.724 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:18.733 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:18.735 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:56.714 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:18.819 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:18.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:18.901 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:18.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:18.902 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:18.902 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:18.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:18.915 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:18.929+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:19.017 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:19.031+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:19.083 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:19.083 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:19.083 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:19.083 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:19.085 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:19.085 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:19.092 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:19.137 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:19.150+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:19.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2023-12-06T09:26:19.219 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:19.219 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:19.219 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:19.219 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:19.219 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:19.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:19.245 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:19.257+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:19.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:26:19.338 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:19.352+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:19.394 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:19.394 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:19.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:19.394 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:19.395 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:19.405 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:19.421 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:19.435+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:19.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:19.527 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:19.541+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:19.571 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:19.571 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:19.572 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:19.572 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:19.585 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:19.626 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:19.640+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:19.726 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:19.740+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:19.764 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:19.764 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:19.764 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:19.765 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:19.765 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:19.776 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:19.827 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:19.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:19.919 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:19.927 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:19.941+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:19.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:20.023 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:20.023 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:20.028 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:20.042+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:20.129 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:20.143+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:20.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:20.230 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:20.244+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:20.244 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:20.244 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:20.277 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:26:20.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2023-12-06T09:26:20.328 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:20.329 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:20.331 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:20.345+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:20.345 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:20.345 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:20.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:20.416 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:20.416 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:20.431 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:20.445+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:20.502 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:20.530 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:20.530 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:20.532 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:20.546+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:20.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:20.600 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:20.633 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:20.647+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:20.715 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:20.715 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:20.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:20.770 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:20.783+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:20.835 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:20.849+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:20.856 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:20.856 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:20.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:20.942 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:20.950+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:21.034 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:21.034 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:21.036 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:21.050+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:21.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:21.137 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:21.151+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:21.217 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:21.217 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:21.229 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:21.238 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:21.252+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:21.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2023-12-06T09:26:21.370 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:21.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:21.440 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:21.454+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:21.444 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:21.542 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:21.556+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:21.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:21.642 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:21.656+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:21.744 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:21.758+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:21.765 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:21.765 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:21.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:21.845 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:21.859+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:21.943 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:21.943 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:21.946 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:21.960+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:21.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:22.047 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:22.061+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:22.120 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:22.121 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:22.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:22.171 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:22.185+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:22.250 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:22.263+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:22.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:26:22.297 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:22.297 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:22.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:22.350 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:22.364+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:22.435 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:22.435 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:22.444 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:22.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2023-12-06T09:26:22.489 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:22.503+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:22.552 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:22.566+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:22.578 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:22.579 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:22.589 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:22.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:22.661 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:22.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:22.754 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:22.768+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:22.757 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:22.757 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:22.766 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:22.857 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:22.870+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:22.895 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:22.895 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:22.905 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:22.907 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:22.907 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:22.989 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:23.003+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:22.997 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:22.997 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:23.058 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:23.071+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:23.075 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:23.075 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:23.084 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:23.085 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:23.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:23.170 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:23.171 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:23.171 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:23.184+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:23.255 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:23.255 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:23.256 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:23.256 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:23.258 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:23.273+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:23.265 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:23.344 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:23.344 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:23.364 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:23.377+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:23.394 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:23.395 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:23.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:23.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2023-12-06T09:26:23.438 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:23.439 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:23.461 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:23.474+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:23.526 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:23.526 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:23.562 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:23.576+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:23.563 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:23.563 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:23.563 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:23.564 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:23.564 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:23.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:23.575 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:26:23.665 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:23.677+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:23.716 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:23.716 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:23.717 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:23.717 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:23.726 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:23.764 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:23.778+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:23.883 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:23.896+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:23.897 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:23.897 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:23.897 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:23.897 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:23.897 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:23.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:23.966 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:23.980+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:24.067 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:24.081+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:24.077 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:24.077 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:24.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:24.077 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:26:24.078 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:24.086 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:24.086 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:24.089 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-12-06T09:26:24.089 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T09:26:24.090 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T09:26:24.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-12-06T09:26:24.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:24.177 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:24.191+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:24.222 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:24.222 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "" 2023-12-06T09:26:24.222 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:24.260 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:24.261 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:24.268 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:24.282+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:24.269 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:24.369 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:24.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:24.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2023-12-06T09:26:24.469 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:24.469 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:24.469 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:24.469 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:24.469 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:24.470 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:24.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:24.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:24.570 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:24.584+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:24.672 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:24.685+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:24.714 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:24.714 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:24.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 267.803751283, 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:24.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:26:24.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:26:24.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:26:24.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 265.38604116599998, 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:26:24.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:26:24.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 265.38185672100002, 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.729 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:26:24.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:26:24.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:26:24.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 265.29632563500002, 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:24.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:26:24.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:26:24.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.743 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.98234902000002, 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:26:24.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:26:24.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:26:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:26:24.748 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.98083117599998, 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.749 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:26:24.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:26:24.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:26:24.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:26:24.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.980165801, 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:26:24.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:26:24.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.763 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.97590562900001, 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.764 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.97527242000001, 2023-12-06T09:26:24.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.770 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:26:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:26:24.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:26:24.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:26:24.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.774 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.776 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.974645318, 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.777 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:26:24.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:26:24.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.88638553999999, 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.782 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:26:24.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.88546456, 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:26:24.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:26:24.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:26:24.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:26:24.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 264.13308662899999, 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:26:24.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:26:24.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:26:24.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.800 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.26305165799999, 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:26:24.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:26:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:26:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.261878147, 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.807 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.25372617599999, 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.812 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:26:24.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:26:24.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.24344423999997, 2023-12-06T09:26:24.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.823 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.24278636000003, 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.24211158100002, 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.829 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:26:24.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 256.20975399000002, 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:26:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:26:24.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:26:24.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:26:24.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:24.839 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:24.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:24.841 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:24.855+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:24.942 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:24.956+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:25.015 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:25.015 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:25.015 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:25.015 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:25.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:25.068 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:25.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:25.144 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:25.158+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:25.154 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:25.154 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:25.154 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:25.154 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:25.154 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:25.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:25.247 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:25.259+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:25.337 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:25.337 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 114.701734676, 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:25.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:25.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 114.396513375, 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:25.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:26:25.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:26:25.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:25.346 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:25.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:25.348 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:25.362+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:25.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2023-12-06T09:26:25.448 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:25.462+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:25.524 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:25.524 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:25.524 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:25.524 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:25.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:25.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:25.550 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:25.564+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:25.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:26:25.652 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:25.665+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:25.673 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:25.673 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:25.673 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:25.673 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:25.673 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:25.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:25.752 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:25.766+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:25.830 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:25.831 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:25.840 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:25.840 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:25.840 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:25.840 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:25.840 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.840 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 250.87616447400001, 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:25.841 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:26:25.842 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:26:25.843 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:25.844 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:25.851 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:25.868 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:25.882+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:25.931 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:25.932 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:25.954 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:25.968+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:26.000 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:26.001 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:26.001 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:26.001 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:26.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:26.027 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:26.028 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:26.059 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:26.073+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:26.134 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:26.134 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:26.160 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:26.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:26.160 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:26.161 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:26.161 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:26.170 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:26.238 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:26.238 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:26.258 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:26.272+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:26.326 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:26.326 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:26.342 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:26.343 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:26.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:26.392 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:26.406+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:26.414 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:26.444 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:26.459 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:26.473+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:26.512 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:26.512 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:26.525 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:26.525 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:26.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:26.568 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:26.581+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:26.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2023-12-06T09:26:26.661 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:26.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:26.702 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:26.702 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:26.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:26.762 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:26.776+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:26.863 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:26.877+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:26.879 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:26.879 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:26.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:26.965 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:26.978+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:27.015 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:27.015 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:27.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:27.066 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:27.079+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:27.167 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:27.181+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:27.195 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:27.195 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:27.202 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:27.268 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:27.282+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:27.369 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:27.369 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:27.369 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:27.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:27.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:27.449 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:26:27.470 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:27.484+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:27.511 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:27.511 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:27.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:27.605 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:27.619+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:27.622 INFO:tasks.mon_recovery:adding mon b back in 2023-12-06T09:26:27.622 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T09:26:27.622 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T09:26:27.624 INFO:tasks.ceph.mon.b:Started 2023-12-06T09:26:27.625 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:26:27.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:27.672 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:27.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:27.694 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:27.694 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:27.704 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:27.775 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:27.789+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:27.876 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:27.890+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:27.876 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:27.877 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:27.888 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:27.979 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:27.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:28.027 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:28.027 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:28.037 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:28.079 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:28.093+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:28.182 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:28.193+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:28.215 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:28.215 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:28.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:28.279 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":35,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:28.283 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:28.297+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:28.387 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:28.401+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:28.395 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:28.395 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:28.405 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:28.489 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:28.503+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:28.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:28.591 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:28.602+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:28.591 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:28.591 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:28.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:28.689 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:28.703+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:28.787 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:28.787 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:28.789 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:28.803+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:28.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:28.859 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:28.890 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:28.904+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:28.949 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:28.991 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:29.005+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:29.038 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:29.038 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 272.13650411499998, 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:26:29.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:26:29.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.054 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.71879399800002, 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:29.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:26:29.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.064 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.714609553, 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.065 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:26:29.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:26:29.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:26:29.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.070 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.629078467, 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:26:29.071 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.076 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.315101852, 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.077 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.082 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:26:29.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.082 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.31358400800002, 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:26:29.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.31291863299998, 2023-12-06T09:26:29.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:26:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:26:29.098 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.30865846099999, 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.099 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.104 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.30802525199999, 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.30739814999998, 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.21913837199997, 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:29.122 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:29.123 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.21821739199999, 2023-12-06T09:26:29.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:26:29.123 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.123 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:26:29.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:26:29.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:26:29.129 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 268.46583946099997, 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.130 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:26:29.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.131 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:26:29.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:26:29.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.135 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.135 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.135 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:26:29.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:26:29.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.136 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.59580448999998, 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.137 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:29.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:26:29.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:26:29.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.142 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.59463097899999, 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:29.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:26:29.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:26:29.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.148 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.58647900800003, 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:26:29.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.57619707200001, 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.57553919200001, 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:26:29.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:26:29.164 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.574864413, 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.165 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:26:29.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.167 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:26:29.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.170 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 260.54250682200001, 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.173 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:29.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.176 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:29.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:29.178 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:29.179 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:29.179 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:29.191+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:29.216 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:29.217 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:29.234 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:29.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:29.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:26:29.302 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:29.303 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:29.317+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:29.348 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:29.349 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:29.349 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:29.349 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:29.358 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:29.380 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:29.394+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:29.401 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:29.401 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:29.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:26:29.482 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:29.496+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:29.488 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:29.488 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:29.558 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:29.558 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:29.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:29.616 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:29.630+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:29.684 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:29.698+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:29.720 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 119.08431662300001, 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:29.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:26:29.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:26:29.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 118.779095322, 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:29.724 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:26:29.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:26:29.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.727 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.728 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.728 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:29.728 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:29.728 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:29.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:29.785 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:29.799+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:29.867 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:26:29.868 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:26:29.883 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2023-12-06T09:26:27.965962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222304,"kb_used_data":5184,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754752416,"statfs":{"total":773094113280,"available":772866473984,"internally_reserved":0,"allocated":5308416,"data_stored":2869965,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.735529"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-06T09:26:17.497017+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:26:17.497017+0000","last_peered":"2023-12-06T09:26:17.497017+0000","last_clean":"2023-12-06T09:26:17.497017+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:26:17.497017+0000","last_undegraded":"2023-12-06T09:26:17.497017+0000","last_fullsized":"2023-12-06T09:26:17.497017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":72,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.572164+0000","last_change":"2023-12-06T09:23:50.752271+0000","last_active":"2023-12-06T09:26:20.572164+0000","last_peered":"2023-12-06T09:26:20.572164+0000","last_clean":"2023-12-06T09:26:20.572164+0000","last_became_active":"2023-12-06T09:23:50.743293+0000","last_became_peered":"2023-12-06T09:23:50.743293+0000","last_unstale":"2023-12-06T09:26:20.572164+0000","last_undegraded":"2023-12-06T09:26:20.572164+0000","last_fullsized":"2023-12-06T09:26:20.572164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:27:50.211606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.284646+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:26:20.284646+0000","last_peered":"2023-12-06T09:26:20.284646+0000","last_clean":"2023-12-06T09:26:20.284646+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:26:20.284646+0000","last_undegraded":"2023-12-06T09:26:20.284646+0000","last_fullsized":"2023-12-06T09:26:20.284646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.568906+0000","last_change":"2023-12-06T09:24:37.205991+0000","last_active":"2023-12-06T09:26:20.568906+0000","last_peered":"2023-12-06T09:26:20.568906+0000","last_clean":"2023-12-06T09:26:20.568906+0000","last_became_active":"2023-12-06T09:24:37.200368+0000","last_became_peered":"2023-12-06T09:24:37.200368+0000","last_unstale":"2023-12-06T09:26:20.568906+0000","last_undegraded":"2023-12-06T09:26:20.568906+0000","last_fullsized":"2023-12-06T09:26:20.568906+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:57:43.491211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":49,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.568665+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:26:20.568665+0000","last_peered":"2023-12-06T09:26:20.568665+0000","last_clean":"2023-12-06T09:26:20.568665+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:26:20.568665+0000","last_undegraded":"2023-12-06T09:26:20.568665+0000","last_fullsized":"2023-12-06T09:26:20.568665+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.563500+0000","last_change":"2023-12-06T09:23:45.621641+0000","last_active":"2023-12-06T09:26:20.563500+0000","last_peered":"2023-12-06T09:26:20.563500+0000","last_clean":"2023-12-06T09:26:20.563500+0000","last_became_active":"2023-12-06T09:23:45.615923+0000","last_became_peered":"2023-12-06T09:23:45.615923+0000","last_unstale":"2023-12-06T09:26:20.563500+0000","last_undegraded":"2023-12-06T09:26:20.563500+0000","last_fullsized":"2023-12-06T09:26:20.563500+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:06:07.386710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":74,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.274521+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:26:20.274521+0000","last_peered":"2023-12-06T09:26:20.274521+0000","last_clean":"2023-12-06T09:26:20.274521+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:26:20.274521+0000","last_undegraded":"2023-12-06T09:26:20.274521+0000","last_fullsized":"2023-12-06T09:26:20.274521+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-12-06T09:26:20.568929+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:26:20.568929+0000","last_peered":"2023-12-06T09:26:20.568929+0000","last_clean":"2023-12-06T09:26:20.568929+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:26:20.568929+0000","last_undegraded":"2023-12-06T09:26:20.568929+0000","last_fullsized":"2023-12-06T09:26:20.568929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":189,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-06T09:26:17.478165+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:26:17.478165+0000","last_peered":"2023-12-06T09:26:17.478165+0000","last_clean":"2023-12-06T09:26:17.478165+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:26:17.478165+0000","last_undegraded":"2023-12-06T09:26:17.478165+0000","last_fullsized":"2023-12-06T09:26:17.478165+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509497,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":260939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":8.7010000000000005,"5min":12.443,"15min":15.491},"min":{"1min":4.71,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":20.803999999999998,"5min":49.691000000000003,"15min":49.691000000000003},"last":6.5389999999999997},{"interface":"front","average":{"1min":8.1069999999999993,"5min":12.114000000000001,"15min":15.531000000000001},"min":{"1min":5.0960000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":12.129,"5min":51.530000000000001,"15min":51.530000000000001},"last":9.3849999999999998}]},{"osd":1,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":10.028,"5min":11.212,"15min":12.114000000000001},"min":{"1min":6.54,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":17.532,"5min":38.488999999999997,"15min":38.488999999999997},"last":5.9320000000000004},{"interface":"front","average":{"1min":10.675000000000001,"5min":13.236000000000001,"15min":14.423},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":16.981999999999999,"5min":35.378999999999998,"15min":35.378999999999998},"last":11.528}]},{"osd":2,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":12.456,"5min":13.795,"15min":15.000999999999999},"min":{"1min":6.9749999999999996,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":23.033000000000001,"5min":42.523000000000003,"15min":42.523000000000003},"last":10.659000000000001},{"interface":"front","average":{"1min":14.18,"5min":16.103000000000002,"15min":17.565999999999999},"min":{"1min":8.1379999999999999,"5min":7.8360000000000003,"15min":7.8360000000000003},"max":{"1min":24.274999999999999,"5min":39.066000000000003,"15min":39.066000000000003},"last":21.725999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":11.548,"5min":12.92,"15min":14.516},"min":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":31.850999999999999,"5min":32.847000000000001,"15min":32.847000000000001},"last":29.457999999999998},{"interface":"front","average":{"1min":14.718,"5min":14.782999999999999,"15min":15.331},"min":{"1min":8.3729999999999993,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":29.949999999999999,"5min":32.104999999999997,"15min":32.104999999999997},"last":31.715}]},{"osd":5,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":19.172999999999998,"15min":24.998999999999999},"min":{"1min":7.1740000000000004,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":21.423999999999999,"5min":190.821,"15min":190.821},"last":22.341000000000001},{"interface":"front","average":{"1min":15.045999999999999,"5min":20.317,"15min":25.199000000000002},"min":{"1min":8.202,"5min":8.202,"15min":8.202},"max":{"1min":28.495000000000001,"5min":191.86199999999999,"15min":191.86199999999999},"last":31.109000000000002}]},{"osd":6,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":15.114000000000001,"5min":20.353000000000002,"15min":24.863},"min":{"1min":8.0210000000000008,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":27.866,"5min":166.45400000000001,"15min":166.45400000000001},"last":28.489000000000001},{"interface":"front","average":{"1min":17.379999999999999,"5min":21.242000000000001,"15min":25.097000000000001},"min":{"1min":10.69,"5min":10.69,"15min":10.69},"max":{"1min":48.956000000000003,"5min":117.58499999999999,"15min":117.58499999999999},"last":27.768000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:24:01 2023","interfaces":[{"interface":"back","average":{"1min":14.968,"5min":18.187999999999999,"15min":18.725000000000001},"min":{"1min":9.4130000000000003,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":20.327999999999999,"5min":32.558,"15min":32.558},"last":23.800999999999998},{"interface":"front","average":{"1min":18.811,"5min":19.652999999999999,"15min":19.792999999999999},"min":{"1min":11.502000000000001,"5min":11.502000000000001,"15min":11.502000000000001},"max":{"1min":31.564,"5min":31.564,"15min":31.564},"last":22.875}]}]},{"osd":2,"up_from":15,"seq":64424509497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":260939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":10.148,"5min":8.9000000000000004,"15min":8.1240000000000006},"min":{"1min":4.0599999999999996,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":26.591000000000001,"5min":38.481000000000002,"15min":38.481000000000002},"last":7.7140000000000004},{"interface":"front","average":{"1min":11.689,"5min":12.025,"15min":12.246},"min":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":25.939,"5min":37.134999999999998,"15min":37.134999999999998},"last":7.1299999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":12.641999999999999,"5min":13.772,"15min":14.842000000000001},"min":{"1min":6.4379999999999997,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":24.552,"5min":52.530000000000001,"15min":52.530000000000001},"last":14.269},{"interface":"front","average":{"1min":17.038,"5min":17.317,"15min":18.417000000000002},"min":{"1min":10.679,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":30.294,"5min":54.338000000000001,"15min":54.338000000000001},"last":15.159000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":10.874000000000001,"5min":11.571,"15min":11.157},"min":{"1min":5.5899999999999999,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":18.417999999999999,"5min":27.295999999999999,"15min":27.295999999999999},"last":11.865},{"interface":"front","average":{"1min":10.398,"5min":11.614000000000001,"15min":12.907999999999999},"min":{"1min":5.4269999999999996,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":28.669,"5min":28.669,"15min":28.669},"last":10.624000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":13.93,"5min":14.285,"15min":14.903},"min":{"1min":7.2130000000000001,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":28.007999999999999,"5min":54.32,"15min":54.32},"last":9.5180000000000007},{"interface":"front","average":{"1min":16.617000000000001,"5min":17.129999999999999,"15min":17.629000000000001},"min":{"1min":9.8780000000000001,"5min":8.1669999999999998,"15min":8.1669999999999998},"max":{"1min":28.863,"5min":51.950000000000003,"15min":51.950000000000003},"last":17.434999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":12.35,"5min":13.502000000000001,"15min":15.311999999999999},"min":{"1min":7.0490000000000004,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":19.962,"5min":56.491,"15min":56.491},"last":13.343},{"interface":"front","average":{"1min":14.957000000000001,"5min":17.225999999999999,"15min":19.25},"min":{"1min":9.8379999999999992,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":31.5,"5min":75.540999999999997,"15min":75.540999999999997},"last":12.553000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":16.591999999999999,"5min":17.579999999999998,"15min":18.154},"min":{"1min":10.010999999999999,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":27.186,"5min":30.213999999999999,"15min":30.213999999999999},"last":19.628},{"interface":"front","average":{"1min":13.185,"5min":13.653,"15min":14.336},"min":{"1min":7.0190000000000001,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":26.312999999999999,"5min":26.312999999999999,"15min":26.312999999999999},"last":18.507999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:23:57 2023","interfaces":[{"interface":"back","average":{"1min":20.181999999999999,"5min":20.227,"15min":20.234999999999999},"min":{"1min":13.214,"5min":13.214,"15min":13.214},"max":{"1min":29.504999999999999,"5min":35.094000000000001,"15min":35.094000000000001},"last":20.655999999999999},{"interface":"front","average":{"1min":15.259,"5min":17.759,"15min":18.175999999999998},"min":{"1min":8.5800000000000001,"5min":8.5800000000000001,"15min":8.5800000000000001},"max":{"1min":26,"5min":27.387,"15min":27.387},"last":21.675999999999998}]}]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":260939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.456,"5min":11.295,"15min":11.247999999999999},"min":{"1min":4.1559999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":65.808000000000007,"5min":65.808000000000007,"15min":65.808000000000007},"last":7.7510000000000003},{"interface":"front","average":{"1min":12.44,"5min":11.872999999999999,"15min":11.510999999999999},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":48.350999999999999,"5min":49.088000000000001,"15min":49.088000000000001},"last":8.9369999999999994}]},{"osd":2,"last update":"Wed Dec 6 09:26:00 2023","interfaces":[{"interface":"back","average":{"1min":13.243,"5min":12.903,"15min":13.025},"min":{"1min":7.5780000000000003,"5min":6.5570000000000004,"15min":6.5570000000000004},"max":{"1min":49.652999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":10.754},{"interface":"front","average":{"1min":13.839,"5min":13.645,"15min":14.175000000000001},"min":{"1min":6.798,"5min":6.798,"15min":6.798},"max":{"1min":47.484000000000002,"5min":47.484000000000002,"15min":47.484000000000002},"last":13.585000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:26:00 2023","interfaces":[{"interface":"back","average":{"1min":11.994999999999999,"5min":15.952,"15min":20.82},"min":{"1min":6.0060000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":41.765999999999998,"5min":205.096,"15min":205.096},"last":32.063000000000002},{"interface":"front","average":{"1min":15.677,"5min":18.654,"15min":22.559999999999999},"min":{"1min":7.0999999999999996,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":52.024999999999999,"5min":182.74000000000001,"15min":182.74000000000001},"last":31.346}]},{"osd":4,"last update":"Wed Dec 6 09:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.57,"5min":13.51,"15min":14.071999999999999},"min":{"1min":6.1189999999999998,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":45.112000000000002,"5min":45.112000000000002,"15min":45.112000000000002},"last":12.987},{"interface":"front","average":{"1min":16.978999999999999,"5min":16.681000000000001,"15min":16.739000000000001},"min":{"1min":8.9290000000000003,"5min":8.4220000000000006,"15min":8.4220000000000006},"max":{"1min":52.655999999999999,"5min":52.655999999999999,"15min":52.655999999999999},"last":25.82}]},{"osd":5,"last update":"Wed Dec 6 09:26:00 2023","interfaces":[{"interface":"back","average":{"1min":14.766,"5min":15.507999999999999,"15min":15.214},"min":{"1min":6.6600000000000001,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":43.088000000000001,"5min":43.088000000000001,"15min":43.088000000000001},"last":11.547000000000001},{"interface":"front","average":{"1min":15.042,"5min":16.381,"15min":16.808},"min":{"1min":8.3659999999999997,"5min":8.266,"15min":8.266},"max":{"1min":45.722000000000001,"5min":45.722000000000001,"15min":45.722000000000001},"last":26.678000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:26:00 2023","interfaces":[{"interface":"back","average":{"1min":18.888999999999999,"5min":19.25,"15min":20.474},"min":{"1min":10.629,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":53.921999999999997,"5min":78.338999999999999,"15min":78.338999999999999},"last":30.710000000000001},{"interface":"front","average":{"1min":17.449000000000002,"5min":18.516999999999999,"15min":20.491},"min":{"1min":10.028,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":50.877000000000002,"5min":80.850999999999999,"15min":80.850999999999999},"last":15.026999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:23:57 2023","interfaces":[{"interface":"back","average":{"1min":17.128,"5min":19.998000000000001,"15min":20.475999999999999},"min":{"1min":8.5619999999999994,"5min":8.5619999999999994,"15min":8.5619999999999994},"max":{"1min":41.692999999999998,"5min":46.590000000000003,"15min":46.590000000000003},"last":9.7840000000000007},{"interface":"front","average":{"1min":16.920000000000002,"5min":20.274000000000001,"15min":20.832999999999998},"min":{"1min":9.7680000000000007,"5min":9.7680000000000007,"15min":9.7680000000000007},"max":{"1min":35.965000000000003,"5min":50.432000000000002,"15min":50.432000000000002},"last":14.443}]}]},{"osd":7,"up_from":60,"seq":257698037767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28612,"kb_used_data":632,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343228,"statfs":{"total":96636764160,"available":96607465472,"internally_reserved":0,"allocated":647168,"data_stored":256530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.614999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.341999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.952}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.416}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.801}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.312999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.442}]}]},{"osd":6,"up_from":14,"seq":60129542200,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":260939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:09 2023","interfaces":[{"interface":"back","average":{"1min":8.8879999999999999,"5min":10.869,"15min":11.734999999999999},"min":{"1min":3.5249999999999999,"5min":3.484,"15min":3.484},"max":{"1min":18.242999999999999,"5min":45.506,"15min":45.506},"last":9.7309999999999999},{"interface":"front","average":{"1min":9.8539999999999992,"5min":11.523999999999999,"15min":12.911},"min":{"1min":6.1829999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":21.774999999999999,"5min":46.450000000000003,"15min":46.450000000000003},"last":10.444000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":13.669,"5min":13.887,"15min":15.307},"min":{"1min":6.766,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":30.434999999999999,"5min":55.398000000000003,"15min":55.398000000000003},"last":7.4740000000000002},{"interface":"front","average":{"1min":12.347,"5min":14.218999999999999,"15min":15.079000000000001},"min":{"1min":6.569,"5min":6.569,"15min":6.569},"max":{"1min":31.419,"5min":41.811999999999998,"15min":41.811999999999998},"last":11.394}]},{"osd":2,"last update":"Wed Dec 6 09:26:09 2023","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":16.809000000000001,"15min":20.863},"min":{"1min":5.8920000000000003,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":17.344000000000001,"5min":208.93799999999999,"15min":208.93799999999999},"last":11.974},{"interface":"front","average":{"1min":12.182,"5min":19.119,"15min":25.003},"min":{"1min":6.0119999999999996,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":26.806000000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":15.526}]},{"osd":3,"last update":"Wed Dec 6 09:26:09 2023","interfaces":[{"interface":"back","average":{"1min":12.474,"5min":15.673999999999999,"15min":17.239999999999998},"min":{"1min":5.8680000000000003,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":18.527999999999999,"5min":43.276000000000003,"15min":43.276000000000003},"last":13.906000000000001},{"interface":"front","average":{"1min":9.8030000000000008,"5min":12.081,"15min":14.513},"min":{"1min":5.7489999999999997,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":21.077000000000002,"5min":41.493000000000002,"15min":41.493000000000002},"last":6.4320000000000004}]},{"osd":4,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":13.486000000000001,"5min":18.669,"15min":23.396999999999998},"min":{"1min":9.9280000000000008,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":27.599,"5min":117.10299999999999,"15min":117.10299999999999},"last":16.536999999999999},{"interface":"front","average":{"1min":14.843999999999999,"5min":20.800000000000001,"15min":26.148},"min":{"1min":5.5990000000000002,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":25.722000000000001,"5min":111.874,"15min":111.874},"last":18.609999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:25:59 2023","interfaces":[{"interface":"back","average":{"1min":10.398999999999999,"5min":13.377000000000001,"15min":16.484000000000002},"min":{"1min":6.8710000000000004,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":17.292000000000002,"5min":39.886000000000003,"15min":39.886000000000003},"last":8.5839999999999996},{"interface":"front","average":{"1min":13.776999999999999,"5min":18.318000000000001,"15min":22.021999999999998},"min":{"1min":9.0009999999999994,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":19.654,"5min":60.530999999999999,"15min":60.530999999999999},"last":17.734000000000002}]},{"osd":7,"last update":"Wed Dec 6 09:23:57 2023","interfaces":[{"interface":"back","average":{"1min":18.222999999999999,"5min":23.966999999999999,"15min":24.923999999999999},"min":{"1min":8.1479999999999997,"5min":8.1479999999999997,"15min":8.1479999999999997},"max":{"1min":73.031000000000006,"5min":87.590999999999994,"15min":87.590999999999994},"last":15.855},{"interface":"front","average":{"1min":13.071,"5min":19.727,"15min":20.837},"min":{"1min":8.3960000000000008,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":18.263999999999999,"5min":50.737000000000002,"15min":50.737000000000002},"last":12.920999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":658779,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":10.07,"5min":11.523999999999999,"15min":13.422000000000001},"min":{"1min":4.1719999999999997,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":40.682000000000002,"5min":45.491,"15min":45.491},"last":6.6100000000000003},{"interface":"front","average":{"1min":10.573,"5min":11.798999999999999,"15min":13.242000000000001},"min":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":39.033999999999999,"5min":49.991999999999997,"15min":49.991999999999997},"last":5.1680000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:26:05 2023","interfaces":[{"interface":"back","average":{"1min":10.128,"5min":16.395,"15min":22.163},"min":{"1min":5.4619999999999997,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":25.672000000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":9.6750000000000007},{"interface":"front","average":{"1min":12.891,"5min":22.353000000000002,"15min":31.279},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":74.992999999999995,"5min":344.79599999999999,"15min":344.79599999999999},"last":9.0690000000000008}]},{"osd":2,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":11.249000000000001,"5min":12.782999999999999,"15min":13.345000000000001},"min":{"1min":6.6820000000000004,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":15.619999999999999,"5min":53.235999999999997,"15min":53.235999999999997},"last":14.145},{"interface":"front","average":{"1min":11.502000000000001,"5min":13.833,"15min":15.202999999999999},"min":{"1min":7.4669999999999996,"5min":7.2359999999999998,"15min":7.2359999999999998},"max":{"1min":17.245000000000001,"5min":54.747,"15min":54.747},"last":12.276999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":16.073,"5min":14.666,"15min":13.891},"min":{"1min":7.8609999999999998,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":55.310000000000002,"5min":55.310000000000002,"15min":55.310000000000002},"last":17.215},{"interface":"front","average":{"1min":12.156000000000001,"5min":14.103,"15min":16.533000000000001},"min":{"1min":5.3760000000000003,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":51.183,"5min":51.183,"15min":51.183},"last":11.625999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":10.711,"5min":16.84,"15min":20.420000000000002},"min":{"1min":6.085,"5min":6.085,"15min":6.085},"max":{"1min":19.465,"5min":281.608,"15min":281.608},"last":9.3770000000000007},{"interface":"front","average":{"1min":14.135,"5min":18.385000000000002,"15min":22.271999999999998},"min":{"1min":9.1460000000000008,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":20.797999999999998,"5min":225.83600000000001,"15min":225.83600000000001},"last":13.529}]},{"osd":6,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":14.204000000000001,"5min":17.994,"15min":21.039999999999999},"min":{"1min":8.1780000000000008,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":23.751999999999999,"5min":198.667,"15min":198.667},"last":15.246},{"interface":"front","average":{"1min":14.598000000000001,"5min":19.044,"15min":22.591999999999999},"min":{"1min":9.2609999999999992,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":23.899000000000001,"5min":208.446,"15min":208.446},"last":16.314}]},{"osd":7,"last update":"Wed Dec 6 09:24:01 2023","interfaces":[{"interface":"back","average":{"1min":15.284000000000001,"5min":24.760999999999999,"15min":26.341000000000001},"min":{"1min":10.561,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":21.145,"5min":229.83500000000001,"15min":229.83500000000001},"last":10.786},{"interface":"front","average":{"1min":16.675999999999998,"5min":23.195,"15min":24.280999999999999},"min":{"1min":11.125,"5min":8.5340000000000007,"15min":8.5340000000000007},"max":{"1min":27.189,"5min":233.59700000000001,"15min":233.59700000000001},"last":22.754000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542200,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":654370,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":8.3879999999999999,"5min":9.4649999999999999,"15min":10.737},"min":{"1min":5.0389999999999997,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":19.157,"5min":40.225999999999999,"15min":40.225999999999999},"last":6.1609999999999996},{"interface":"front","average":{"1min":7.9000000000000004,"5min":10.471,"15min":11.545999999999999},"min":{"1min":4.6280000000000001,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":13.375999999999999,"5min":30.405000000000001,"15min":30.405000000000001},"last":7.1749999999999998}]},{"osd":2,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":12.644,"5min":13.731999999999999,"15min":11.704000000000001},"min":{"1min":5.8879999999999999,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":44.145000000000003,"5min":162.74299999999999,"15min":162.74299999999999},"last":9.2850000000000001},{"interface":"front","average":{"1min":15.718999999999999,"5min":17.219999999999999,"15min":16.044},"min":{"1min":7.1769999999999996,"5min":6.415,"15min":6.415},"max":{"1min":42.116999999999997,"5min":164.911,"15min":164.911},"last":10.004}]},{"osd":3,"last update":"Wed Dec 6 09:26:04 2023","interfaces":[{"interface":"back","average":{"1min":12.239000000000001,"5min":16.550999999999998,"15min":20.359000000000002},"min":{"1min":8.0980000000000008,"5min":5.8230000000000004,"15min":5.8230000000000004},"max":{"1min":18.734000000000002,"5min":122.762,"15min":122.762},"last":7.9139999999999997},{"interface":"front","average":{"1min":10.1,"5min":15.638,"15min":19.683},"min":{"1min":6.2249999999999996,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":19.849,"5min":129.99100000000001,"15min":129.99100000000001},"last":18.242999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":13.654999999999999,"5min":21.774999999999999,"15min":29.103999999999999},"min":{"1min":7.0149999999999997,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":29.856000000000002,"5min":359.46300000000002,"15min":359.46300000000002},"last":17.457000000000001},{"interface":"front","average":{"1min":14.845000000000001,"5min":20.852,"15min":27.786999999999999},"min":{"1min":7.3129999999999997,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":34.185000000000002,"5min":314.47699999999998,"15min":314.47699999999998},"last":12.021000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":15.169,"5min":22.873999999999999,"15min":26.449999999999999},"min":{"1min":10.288,"5min":9.9770000000000003,"15min":9.9770000000000003},"max":{"1min":27.010999999999999,"5min":152.97399999999999,"15min":152.97399999999999},"last":10.946},{"interface":"front","average":{"1min":13.765000000000001,"5min":20.765000000000001,"15min":24.774000000000001},"min":{"1min":8.375,"5min":8.375,"15min":8.375},"max":{"1min":23.766999999999999,"5min":154.53100000000001,"15min":154.53100000000001},"last":13.968999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":15.954000000000001,"15min":17.277000000000001},"min":{"1min":7.8410000000000002,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":26.643999999999998,"5min":43.045000000000002,"15min":43.045000000000002},"last":12.788},{"interface":"front","average":{"1min":16.457999999999998,"5min":18.988,"15min":20.623999999999999},"min":{"1min":9.1920000000000002,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":29.367000000000001,"5min":41.356000000000002,"15min":41.356000000000002},"last":16.692}]},{"osd":7,"last update":"Wed Dec 6 09:23:55 2023","interfaces":[{"interface":"back","average":{"1min":17.962,"5min":31.026,"15min":33.203000000000003},"min":{"1min":12.875999999999999,"5min":12.875999999999999,"15min":12.875999999999999},"max":{"1min":43.027000000000001,"5min":285.81700000000001,"15min":285.81700000000001},"last":15.859},{"interface":"front","average":{"1min":17.516999999999999,"5min":30.350999999999999,"15min":32.490000000000002},"min":{"1min":8.8610000000000007,"5min":8.8610000000000007,"15min":8.8610000000000007},"max":{"1min":44.119999999999997,"5min":295.28500000000003,"15min":295.28500000000003},"last":14.627000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":256530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":9.9190000000000005,"5min":12.819000000000001,"15min":13.999000000000001},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":20.942,"5min":137.13999999999999,"15min":137.13999999999999},"last":9.9870000000000001},{"interface":"front","average":{"1min":8.0670000000000002,"5min":14.938000000000001,"15min":18.399999999999999},"min":{"1min":4.7939999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":22.126999999999999,"5min":199.268,"15min":199.268},"last":8.9700000000000006}]},{"osd":1,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":15.423,"15min":18.928999999999998},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":30.039000000000001,"5min":92.817999999999998,"15min":92.817999999999998},"last":11.461},{"interface":"front","average":{"1min":12.808,"5min":16.564,"15min":20.143999999999998},"min":{"1min":5.556,"5min":5.556,"15min":5.556},"max":{"1min":31.067,"5min":95.665999999999997,"15min":95.665999999999997},"last":12.750999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":13.289,"5min":20.170999999999999,"15min":23.940000000000001},"min":{"1min":7.3369999999999997,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":22.687999999999999,"5min":208.06,"15min":208.06},"last":14.661},{"interface":"front","average":{"1min":12.989000000000001,"5min":17.562999999999999,"15min":19.035},"min":{"1min":8.0289999999999999,"5min":6.915,"15min":6.915},"max":{"1min":23.326000000000001,"5min":147.18199999999999,"15min":147.18199999999999},"last":15.846}]},{"osd":3,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.558,"15min":18.673999999999999},"min":{"1min":5.7279999999999998,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":27.295999999999999,"5min":121.627,"15min":121.627},"last":12.050000000000001},{"interface":"front","average":{"1min":16.620000000000001,"5min":17.239999999999998,"15min":18.469999999999999},"min":{"1min":10.382,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":38.029000000000003,"5min":74.602000000000004,"15min":74.602000000000004},"last":16.556999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":15.308,"5min":17.649999999999999,"15min":19.106999999999999},"min":{"1min":11.773,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":24.001999999999999,"5min":71.759,"15min":71.759},"last":18.925999999999998},{"interface":"front","average":{"1min":11.436,"5min":15.087999999999999,"15min":17.015999999999998},"min":{"1min":6.734,"5min":6.6760000000000002,"15min":6.6760000000000002},"max":{"1min":21.036999999999999,"5min":70.882999999999996,"15min":70.882999999999996},"last":13.992000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:26:02 2023","interfaces":[{"interface":"back","average":{"1min":14.936999999999999,"5min":17.821000000000002,"15min":20.337},"min":{"1min":6.6639999999999997,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":48.136000000000003,"5min":80.668999999999997,"15min":80.668999999999997},"last":15.257},{"interface":"front","average":{"1min":19.364000000000001,"5min":20.213000000000001,"15min":22.120999999999999},"min":{"1min":13.02,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":46.707000000000001,"5min":72.796999999999997,"15min":72.796999999999997},"last":19.864999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:23:56 2023","interfaces":[{"interface":"back","average":{"1min":17.483000000000001,"5min":26.989000000000001,"15min":28.573},"min":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":9.0389999999999997},"max":{"1min":42.055,"5min":207.19399999999999,"15min":207.19399999999999},"last":18},{"interface":"front","average":{"1min":19.245999999999999,"5min":27.547999999999998,"15min":28.931999999999999},"min":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"max":{"1min":36.085000000000001,"5min":208.68000000000001,"15min":208.68000000000001},"last":20.666}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:26:29.887 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:29.901+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:29.905 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:29.905 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:29.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:30.000 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:30.014+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:30.089 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:30.103+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:30.103 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:30.103 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:30.103 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:30.103 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:30.103 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:30.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:30.199 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:30.212+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 255.32823909300001, 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:26:30.290 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:30.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:30.293 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:30.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:30.294 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:30.294 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:30.294 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:30.304+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:30.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:30.392 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:30.406+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:30.431 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:30.431 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:30.431 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:30.431 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:30.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:30.493 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:30.507+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:30.572 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:30.573 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:30.573 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:30.573 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:30.573 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:30.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:30.594 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:30.608+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:30.694 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:30.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:30.712 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:30.712 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:30.712 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:30.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:30.795 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:30.809+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:30.851 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:30.852 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:30.852 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:30.852 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:30.860 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:26:30.861 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:30.915 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:30.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:30.990 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:30.991 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:30.991 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:30.991 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:30.991 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:30.997 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:31.012+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:31.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:31.114 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:31.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:31.169 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:31.169 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 272.46311382200003, 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:31.170 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:31.171 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:31.172 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:31.173 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.768659953, 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.174 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.175 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.176 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:31.177 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.325767477, 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.178 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.179 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:31.180 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:31.181 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.31694119600002, 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.182 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.184 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.185 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.31631952399999, 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.187 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:31.188 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.188 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.188 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.188 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.188 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:31.188 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:31.189 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:31.190 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:31.191 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.30837304300002, 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.192 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.193 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.194 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.195 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.30789902499998, 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.196 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:31.197 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:31.198 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:31.199 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 271.30744467, 2023-12-06T09:26:31.200 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.201 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.202 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.203 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.204 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.923452419, 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.205 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.206 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.206 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:31.206 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.206 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.206 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.206 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:31.207 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.208 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:31.209 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:31.210 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.53830301400001, 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:31.212 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:31.213 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.214 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.52602770300001, 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:31.215 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.216 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.217 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.218 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 270.52454212499998, 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.219 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.220 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.221 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.222 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.223 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.61800496699999, 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.224 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.225 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.226 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.227 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:31.228 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.61287266400001, 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.229 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.230 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.231 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:31.232 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.61204663400002, 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.233 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.234 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.235 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:31.236 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.60947717300002, 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.237 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.238 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:31.239 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.240 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.60873170299999, 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.241 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.242 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.243 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.244 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.60792563199999, 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.245 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.246 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:31.247 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:31.248 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 262.59682118000001, 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.249 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.250 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:31.251 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:31.252 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 113.823489043, 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.254 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:31.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:31.257 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:31.258 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:31.259 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:31.272+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:31.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:31.360 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:31.373+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:31.460 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:31.461 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:31.461 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:31.461 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:31.462 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:31.474+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:31.470 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:31.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:31.562 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:31.576+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:31.658 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:31.658 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:31.659 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:31.659 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:31.663 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:31.677+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:31.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:31.764 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:31.777+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:31.837 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:31.837 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:31.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:31.837 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:31.837 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:31.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:31.846 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:31.866 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:31.878+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:31.932 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:31.932 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:31.947 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":39,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:31.967 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:31.981+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:31.974 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:31.974 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:31.974 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:31.974 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:31.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:32.023 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:32.069 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:32.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:32.111 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:32.174 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:32.175 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:32.175 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:32.175 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:32.175 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:32.184 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:32.213 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:32.226+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:32.213 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:32.214 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:32.269 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:32.283+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:32.306 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:32.344 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:32.344 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:32.344 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:32.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:32.370 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:32.384+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:32.393 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:32.471 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:32.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:32.481 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:32.485 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:32.486 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:32.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:32.572 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:32.586+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:32.662 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:32.662 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:32.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:32.673 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:32.687+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:32.774 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:32.788+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:32.804 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:32.804 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:32.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:32.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:26:32.875 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:32.889+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:32.947 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:32.947 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:32.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:33.001 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:33.015+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:33.077 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:33.091+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:33.089 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:33.089 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:33.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:33.183 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:33.197+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:33.271 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:33.272 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:33.278 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:33.292+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:33.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:33.379 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:33.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:33.451 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:33.451 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:33.462 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:33.509 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:33.523+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:33.581 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:33.595+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:33.600 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:33.600 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:33.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:33.682 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:33.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:33.700 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:30.763 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:33.702 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:30.764 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:33.721 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:30.782 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:33.753 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:11.731 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:33.754 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:11.733 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:33.772 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:11.751 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:33.783 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:33.797+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:33.784 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:33.784 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:33.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:33.884 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:33.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:33.944 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:33.944 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:33.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:33.997 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:34.011+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:34.060 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:26:34.087 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:34.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:34.100 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:34.100 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:34.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:34.188 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:34.202+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:34.289 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:34.303+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:34.314 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:34.315 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:34.324 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:34.390 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:34.404+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:34.457 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:34.457 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:34.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:34.492 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:34.506+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:34.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:34.593 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:34.607+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:34.636 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:34.636 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:34.645 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:34.694 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:34.708+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:34.727 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:34.794 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:34.795 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:34.795 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:34.809+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:34.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:34.812 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:34.812 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:34.887 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:34.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:34.887 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-12-06T09:26:34.887 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.470847 2023-12-06T09:26:34.888 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0.4708467256901592 2023-12-06T09:26:34.895 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:34.895 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:34.896 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:34.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:34.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:34.972 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:34.973 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:34.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:35.023 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:35.037+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:35.025 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:35.025 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:35.106 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:35.120+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:35.127 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:35.127 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:35.180 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:35.180 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:35.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:35.198 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:35.212+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:35.241 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:35.241 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:35.299 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:35.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:35.328 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:35.328 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:35.379 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:35.379 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:35.379 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:35.379 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 276.672140533, 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.380 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:35.381 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.382 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.383 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.97768666399998, 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.384 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:35.385 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.387 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:35.388 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.53479418799998, 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.389 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.390 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.391 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:35.392 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.52596790699999, 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.394 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.395 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.396 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.52534623499997, 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.397 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:35.398 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:35.399 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:35.400 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.517399754, 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:35.401 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:35.402 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:35.403 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:35.404 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.51692573600002, 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:35.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.51647138099997, 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.409 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:35.410 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:35.411 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:35.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.13247912999998, 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.413 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:35.414 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:35.415 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 274.74732972499999, 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:35.417 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.418 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.419 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.420 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.421 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 274.73505441399999, 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.422 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.423 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:35.423 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.423 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.423 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.423 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.424 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.425 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.426 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 274.73356883600002, 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.427 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.428 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.430 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.82703167800003, 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.431 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.432 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.433 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.434 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.82189937499999, 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.435 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.436 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.437 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.438 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.821073345, 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.439 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:35.440 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.441 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.442 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.81850388399999, 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.443 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.444 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.444 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.444 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:35.444 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.444 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:35.445 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:35.446 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.447 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.81775841400002, 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.448 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.449 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.450 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.451 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.81695234300003, 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.452 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:35.453 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.454 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 266.80584789099998, 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.456 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.457 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.458 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.459 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 118.032515754, 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:35.461 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.462 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.463 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:35.464 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:35.464 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:35.464 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:35.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:35.466 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:35.479+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:35.466 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:35.566 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:35.580+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:35.636 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:35.636 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:35.636 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:35.636 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:35.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:35.667 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:35.681+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:35.756 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:08.071 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:35.770 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:35.784+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:35.775 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:35.775 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:35.775 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:35.775 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:35.775 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:35.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:35.871 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:35.885+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:35.877 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":43,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:35.916 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:35.916 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:35.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:35.917 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:35.917 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:35.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:35.999 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:08.314 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:36.000 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:36.014+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:36.072 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:36.086+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:36.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:26:36.099 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:36.099 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:36.099 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:36.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:36.193 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:36.207+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:36.274 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:36.288+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:36.279 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:36.279 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:36.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:36.280 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:36.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:36.375 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:36.389+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:36.459 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:36.459 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:36.459 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:36.459 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:36.459 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:36.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:36.509 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:36.523+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:36.547 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.0 primary-affinity to 0.470847 (8308572) 2023-12-06T09:26:36.577 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:36.590+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:36.626 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:36.627 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:36.627 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:36.627 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:36.636 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:36.678 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:36.691+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:36.778 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:36.792+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:36.805 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:36.805 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:36.805 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:36.806 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:36.806 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:36.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:36.879 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:36.893+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:36.980 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:36.994+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:36.985 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:36.985 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:36.985 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:36.985 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:26:36.985 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:36.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:37.081 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:37.095+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:37.169 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:37.169 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:37.170 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:37.170 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:37.176 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:37.218 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:37.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:37.283 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:37.296+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:37.344 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:37.344 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:37.344 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:37.344 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:37.344 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:37.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:37.397 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:37.411+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:37.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:37.485 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:37.499+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:37.550 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 280.63962532400001, 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:26:37.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:26:37.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.556 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 278.221915207, 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:26:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:26:37.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:26:37.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:26:37.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 278.21773076199997, 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.562 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:37.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:26:37.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:26:37.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:26:37.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 278.13219967600003, 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:26:37.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:26:37.572 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.81822306100003, 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:26:37.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:26:37.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:26:37.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:26:37.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:26:37.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.81670521699999, 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.81603984200001, 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.584 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.81177967000002, 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:26:37.589 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:26:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.81114646100002, 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:26:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:26:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:26:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.599 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.81051935900001, 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.600 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.722259581, 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.605 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.72133860100001, 2023-12-06T09:26:37.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 276.96896067, 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.09892569900001, 2023-12-06T09:26:37.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:26:37.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:26:37.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:26:37.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.627 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.09775218800002, 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:37.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:26:37.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:26:37.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.089600217, 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:26:37.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:26:37.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:26:37.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:26:37.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.07931828099998, 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.638 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:37.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:26:37.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.07866040099998, 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:26:37.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:26:37.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:26:37.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.648 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.07798562200003, 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:26:37.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:26:37.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:26:37.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 269.04562803099998, 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:37.654 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:37.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:37.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:37.659 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:37.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:37.661 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:37.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:37.674 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:26:37.762 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:37.776+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:37.820 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:37.821 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:37.827 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:37.828 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:37.828 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:37.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:37.862 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:37.876+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:37.908 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:37.909 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:37.969 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:37.982+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:38.014 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:38.014 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:38.017 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.017 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:38.027 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:38.068 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:38.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:38.109 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:38.109 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:38.166 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:38.180+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:38.197 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.197 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:38.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:38.213 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:38.214 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:38.267 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:38.281+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:38.344 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:38.344 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:38.352 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.353 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:38.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:38.403 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:38.417+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:38.429 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:38.429 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:38.470 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:38.484+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:38.530 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:38.530 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:38.530 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.530 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:38.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:38.571 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:38.585+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:38.672 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:38.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:38.717 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.717 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:38.727 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:38.773 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:38.787+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:38.854 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.854 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:38.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:38.875 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:38.889+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:38.890 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:38.975 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:38.989+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:38.990 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:38.991 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:39.076 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:39.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:39.128 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:39.128 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.137 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:39.179 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:39.192+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:39.278 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:39.292+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:39.307 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:39.308 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:39.379 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:39.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:39.447 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:39.447 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:39.506 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:39.520+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:39.516 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":46,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:39.580 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:39.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:39.591 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:39.591 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.602 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:39.686 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:39.700+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:39.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:26:39.772 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:39.773 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.782 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:39.783 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:39.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:39.883 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:39.897+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:39.948 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:39.948 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:39.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:40.001 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:40.015+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:40.084 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:40.084 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:40.085 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:40.099+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:40.093 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:40.186 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:40.200+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:40.262 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:40.262 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:40.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:40.287 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:40.301+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:40.388 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:40.402+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:40.440 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:40.440 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:40.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:40.491 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:40.505+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:40.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:40.591 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:40.605+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:40.619 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:40.619 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:40.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:40.695 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:40.708+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:40.795 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:40.809+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:40.796 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:40.796 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:40.797 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:40.797 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:40.797 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:40.797 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:40.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:40.898 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:40.898 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:40.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:40.899 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:40.957 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:26:40.965 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:40.965 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:40.965 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:40.966 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:40.966 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:40.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:40.997 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:41.011+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:41.001 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:41.092 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:41.092 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:41.098 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:41.112+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:41.136 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:41.136 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:41.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:41.136 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:26:41.136 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:41.145 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:41.178 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:41.205 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:41.218+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:41.264 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:41.264 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:41.299 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:41.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:41.307 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:41.307 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:41.307 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:41.307 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:41.317 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:41.348 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:41.349 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:41.400 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:41.414+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:41.433 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:41.433 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:41.479 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:41.479 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:41.480 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:41.480 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:41.480 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:41.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:41.531 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:41.545+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:41.567 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:41.568 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:41.601 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:41.615+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:41.702 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:41.716+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:41.716 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 284.80932726399999, 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 282.39161714699998, 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.723 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.724 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:26:41.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.725 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:26:41.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:26:41.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:26:41.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:26:41.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 282.38743270200001, 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:26:41.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:26:41.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:26:41.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:26:41.737 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 282.30190161600001, 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.738 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:26:41.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:26:41.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.98792500100001, 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:26:41.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:26:41.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.98640715699997, 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:26:41.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.98574178199999, 2023-12-06T09:26:41.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:26:41.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.98148161, 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.762 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:26:41.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.766 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.980848401, 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.768 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:26:41.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.773 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.98022129899999, 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.774 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:26:41.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:26:41.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.780 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.89196152099998, 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.781 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:26:41.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:26:41.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.891040541, 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:26:41.787 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:41.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:26:41.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:26:41.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:26:41.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:26:41.792 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 281.13866260999998, 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.793 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:26:41.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:26:41.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.26862763899999, 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.798 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:26:41.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:41.803 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.267454128, 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:26:41.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:26:41.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.808 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.25930215699998, 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:26:41.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:26:41.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.24902022100002, 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.814 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.819 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.24836234100002, 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.24768756200001, 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.825 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:26:41.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:26:41.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:26:41.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:41.831 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 273.21532997100002, 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:41.832 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:26:41.833 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:41.834 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:26:41.835 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:41.836 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:41.837 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:41.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:41.839 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:41.853+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:41.939 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:41.953+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:42.014 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:42.014 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:42.014 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:42.015 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:42.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:42.068 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:42.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:42.141 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:42.155+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:42.153 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:42.153 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:42.153 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:42.153 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:42.153 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:42.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:42.246 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:42.260+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:42.335 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 131.69934087999999, 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:42.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:26:42.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:26:42.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 131.39411957900001, 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:26:42.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:26:42.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:42.344 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:42.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:42.347 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:42.360+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:42.448 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:42.461+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:42.518 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:42.518 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:42.518 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:42.518 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:42.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:42.531 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:42.571 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:42.585+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:42.650 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:42.663+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:42.703 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:42.703 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:42.703 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:42.703 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:42.703 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:42.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:42.750 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:42.764+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:42.852 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:42.866+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:42.885 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:42.885 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:42.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 267.92392099199998, 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:26:42.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:26:42.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:26:42.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:26:42.889 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:26:42.890 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:42.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:42.953 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:42.966+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:42.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:26:43.028 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:43.028 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:43.029 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:43.029 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:43.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:43.054 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:43.068+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:43.155 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:43.169+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:43.168 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:43.168 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:43.168 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:43.168 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:43.168 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:43.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:43.256 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:43.270+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:43.306 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:43.307 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:43.307 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:43.307 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:43.307 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:43.316 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:43.358 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:43.372+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:43.445 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":50,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:43.447 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:43.448 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:43.448 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:43.448 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:43.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:43.460 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:43.473+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:43.571 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:43.585+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:43.585 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:43.586 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:43.586 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:43.586 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:43.586 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:43.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:43.695 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:43.695 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:43.696 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:43.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:43.738 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:40.800 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:43.742 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:40.805 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:43.757 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:40.819 107113== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:43.760 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:43.774+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:43.788 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:21.766 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:43.791 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:21.769 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:43.803 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:43.803 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:43.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:43.803 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:43.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 285.09819261400003, 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:43.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:43.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 284.403738745, 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:43.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:43.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:43.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.811 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.960846269, 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.815 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.95201998800002, 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.819 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.951398316, 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.94345183500002, 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:43.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:43.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.827 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.94297781699998, 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:43.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:43.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.831 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.942523462, 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:43.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:43.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:43.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:43.835 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.558531211, 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:43.839 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.17338180600001, 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.16110649500001, 2023-12-06T09:26:43.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:43.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:43.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.847 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 283.15962091699998, 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:43.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:43.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:43.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.851 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.25308375899999, 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:43.855 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.24795145600001, 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:43.859 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.24712542600003, 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.860 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.861 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.862 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.24455596500002, 2023-12-06T09:26:43.863 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.864 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:43.865 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:43.866 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.24381049499999, 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:43.867 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.868 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:43.869 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:43.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:43.871 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.24300442399999, 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.872 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.873 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.874 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 275.23189997200001, 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.876 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.877 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:43.878 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 126.458567835, 2023-12-06T09:26:43.879 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:43.880 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:43.882 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:43.883 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:43.884 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:43.884 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:43.885 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:21.785 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:43.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:43.886 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:43.899+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:43.907 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:43.907 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:43.986 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:44.000+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:44.006 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:44.007 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:44.053 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:44.054 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:44.054 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:44.054 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:44.063 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:44.087 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:44.101+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:44.096 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:44.096 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:44.157 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:26:44.194 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:44.194 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:44.194 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:44.194 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:44.194 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:44.199 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:44.199 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:44.199 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:44.213+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:44.204 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:44.289 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:44.302+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:44.298 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:44.339 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:44.340 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:44.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:44.385 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:44.386 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:44.391 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:44.405+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:44.505 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:44.519+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:44.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:44.591 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:44.605+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:44.692 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:44.706+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:44.698 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:44.698 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:44.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:44.793 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:44.807+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:44.839 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:44.839 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:44.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:44.894 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:44.908+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:44.985 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:44.986 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:44.995 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:44.996 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:45.008+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:45.096 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:45.110+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:45.168 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:45.168 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:45.177 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:45.198 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:45.212+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:45.299 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:45.313+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:45.310 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:45.311 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:45.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:45.403 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:45.416+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:45.459 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:17.774 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:45.490 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:45.490 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:45.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:45.503 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:45.517+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:45.511 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:17.826 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:45.607 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:45.621+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:45.627 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:45.627 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:45.636 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:45.708 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:45.722+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:45.757 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:18.071 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:26:45.804 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:45.804 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:45.810 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:45.824+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:45.813 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:45.911 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:45.925+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:45.983 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:45.983 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:45.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:46.012 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:46.025+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:46.113 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:46.127+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:46.122 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:46.122 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:46.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:46.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:26:46.221 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:46.234+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:46.268 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:46.268 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:46.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:46.314 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:46.328+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:46.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:46.443 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:46.457+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:46.459 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:46.471 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:46.472 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:46.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:46.526 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:46.539+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:46.568 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:46.569 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:46.569 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2023-12-06T09:26:46.569 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T09:26:46.569 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T09:26:46.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2023-12-06T09:26:46.616 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:46.630+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:46.677 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:46.700 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:46.700 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T09:26:46.700 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:46.703 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:46.703 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:46.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:46.757 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:46.771+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:46.783 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:46.818 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:46.832+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:46.876 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:46.876 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:46.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:46.915 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:46.915 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:46.919 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:46.933+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:47.012 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:47.013 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:47.013 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:47.013 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:47.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:47.045 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:47.059+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:47.106 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:47.106 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:47.121 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:47.135+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:47.180 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:47.180 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:47.180 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:47.180 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:47.180 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:47.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:47.219 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:47.231 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:26:47.232 INFO:tasks.mon_recovery:removing mon c 2023-12-06T09:26:47.232 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T09:26:47.232 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:26:47.232 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:47.232 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:47.234 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:47.248+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:47.286 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:26:47.290+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 107108) UID: 0 2023-12-06T09:26:47.286 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:26:47.291+0000 9f54640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2023-12-06T09:26:47.317 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:47.317 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:47.318 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:47.318 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:47.318 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:47.318 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:47.318 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:47.322 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:47.337+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:47.328 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:47.406 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:47.406 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:47.424 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:47.438+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:47.518 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:47.519 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:47.519 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:47.519 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:47.525 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:47.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:47.528 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:47.625 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:47.639+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:47.699 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:47.699 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:47.699 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:47.699 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:47.699 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:47.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:47.752 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:47.766+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:47.827 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:47.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 289.19088931800002, 2023-12-06T09:26:47.896 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.897 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.898 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.899 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.49643544899999, 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:26:47.901 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:26:47.902 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.903 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.904 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.905 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.05354297299999, 2023-12-06T09:26:47.906 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.907 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.908 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.909 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.910 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.04471669200001, 2023-12-06T09:26:47.911 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.912 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.913 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:26:47.914 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.915 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.04409501999999, 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.916 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.917 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:26:47.918 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.919 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:26:47.920 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:26:47.922 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.03614853900001, 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:26:47.924 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:26:47.925 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.926 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.03567452099998, 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.927 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.928 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.929 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:26:47.930 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 288.03522016599999, 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.931 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:26:47.932 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:26:47.933 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.934 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 287.65122791499999, 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.935 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:26:47.936 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:26:47.937 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:26:47.938 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 287.26607851, 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.939 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:26:47.940 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:26:47.941 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:26:47.942 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 287.253803199, 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:26:47.943 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.944 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:26:47.945 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.946 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 287.25231762099997, 2023-12-06T09:26:47.947 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.948 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:26:47.949 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:26:47.950 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:26:47.951 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.34578046299998, 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:47.952 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:26:47.953 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:26:47.954 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:26:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.34064816, 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.956 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.957 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.958 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.959 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.33982213000002, 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.960 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:26:47.961 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:26:47.962 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:26:47.963 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.33725266900001, 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.964 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:26:47.965 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:26:47.966 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:26:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.33650719899998, 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.968 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:26:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:26:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:26:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.33570112799998, 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:26:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.974 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 279.324596676, 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.976 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.977 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.978 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.979 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 130.55126453899999, 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:26:47.980 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.981 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.982 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.983 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.984 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:26:47.984 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:26:47.984 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:47.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:47.986 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:47.999+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:48.024 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:45.086 107113== 2023-12-06T09:26:48.024 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:45.086 107113== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:26:48.036 INFO:tasks.ceph.mon.c.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:26:48.086 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:48.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:48.154 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:48.154 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:48.154 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:48.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:48.207 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:48.221+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:48.288 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:48.302+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:48.289 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:48.289 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:48.289 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:48.290 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:48.290 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:48.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:48.389 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:48.403+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:48.469 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:48.469 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:48.469 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:48.469 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:48.469 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:48.478 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:48.520 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:48.534+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:48.591 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:48.605+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:48.647 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:48.647 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:26:48.647 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:48.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:48.692 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:48.706+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:48.793 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:48.807+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:48.824 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:48.824 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:26:48.824 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:26:48.824 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:26:48.824 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:48.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:48.918 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:48.932+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:48.995 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:49.009+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:49.001 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:26:49.001 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:26:49.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:26:49.002 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:26:49.002 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:26:49.010 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:49.096 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:49.110+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:49.164 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:49.164 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:49.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:49.217 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:49.231+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:49.297 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:49.311+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:49.302 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:49.302 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:49.302 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:26:49.302 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:26:49.302 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:49.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:49.398 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:49.412+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:49.421 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:26:49.497 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:49.497 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:26:49.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:26:49.497 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:26:49.498 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:49.499 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:49.513+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:49.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:49.600 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:49.614+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:49.690 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:26:49.690 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:26:49.690 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:26:49.690 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:26:49.691 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:26:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:26:49.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:49.701 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:49.714+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:49.778 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:26:49.779 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:26:49.801 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:49.815+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:49.842 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:49.843 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:49.852 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:49.878 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:26:49.879 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:26:49.923 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:49.937+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:49.972 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:26:49.972 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:26:50.003 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:50.016+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:50.009 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:50.009 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:50.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:50.057 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:26:50.058 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:26:50.142 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:26:50.143 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:26:50.143 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:50.156+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:50.189 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:50.189 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:50.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:50.204 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:50.218+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:50.227 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:26:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:26:50.310 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:26:50.311 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:50.324+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:50.311 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:26:50.328 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:50.328 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:50.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:50.406 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:50.420+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:50.507 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:50.507 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:50.507 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:50.521+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:50.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:50.608 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:50.622+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:50.685 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:50.686 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:50.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:50.709 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:50.723+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:50.810 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:50.824+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:50.823 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:50.823 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:50.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:50.911 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:50.925+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:51.002 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.002 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:51.013 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:51.026+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:51.113 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:51.127+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:51.144 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.144 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.154 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:51.214 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:51.228+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:51.282 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.283 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.292 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:51.336 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:51.350+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:51.415 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:51.429+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:51.421 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.421 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:51.516 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:51.530+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:51.599 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.599 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:51.617 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:51.631+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:51.711 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:51.711 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:26:51.718 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:51.731+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:51.781 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.781 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:51.819 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:51.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:51.921 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:51.933+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:51.961 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:51.961 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:51.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:52.055 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:52.069+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:52.125 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:52.138+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:52.139 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:52.139 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:52.149 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:52.222 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:52.236+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:52.315 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:52.315 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:52.323 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:52.337+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:52.324 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:52.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:26:52.424 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:52.437+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:52.454 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:52.454 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:52.464 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:52.524 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:52.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:52.593 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:52.593 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:52.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:52.646 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:52.660+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:52.726 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:52.740+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:52.729 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:52.730 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:52.739 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:52.827 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:52.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:52.911 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:52.911 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:52.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:52.928 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:52.942+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:53.029 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:53.043+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:53.052 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:53.052 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:53.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T09:26:53.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:53.146 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:53.160+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:53.230 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:53.231 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:53.231 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:53.245+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:53.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:53.332 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:53.346+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:26:53.334 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T09:26:53.334 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T09:26:53.334 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T09:26:53.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:53.411 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:53.412 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:53.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:53.433 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:53.448+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:53.538 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:53.552+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:53.553 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:53.553 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:53.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:53.648 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:53.662+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:53.736 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:53.750+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:53.741 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:53.741 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:53.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:53.837 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:53.852+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:53.918 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:53.919 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:53.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:53.972 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:53.986+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:53.985 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:26:54.039 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:54.053+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:54.062 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:54.062 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:54.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:54.140 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:54.154+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:54.239 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:54.239 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:54.241 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:54.254+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:54.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:54.344 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:54.355+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:54.417 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:54.417 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:54.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:54.442 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:54.457+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:54.543 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:54.557+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:54.594 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:54.595 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:54.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:54.644 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:54.658+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:54.745 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:54.759+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:54.772 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:54.772 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:54.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:54.846 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:54.860+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:54.909 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:54.909 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:54.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:54.961 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:54.974+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:55.049 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:55.063+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:55.060 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:55.060 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:55.071 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:55.150 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:55.163+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:55.200 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:55.201 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:55.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:55.254 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:55.268+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:55.337 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:55.337 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:55.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:55.351 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:55.365+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:55.453 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:55.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:55.515 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:55.515 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:55.524 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:55.554 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:55.567+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:55.654 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:55.668+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:55.693 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:55.693 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:55.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:55.756 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:55.769+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:55.833 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:55.833 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:55.842 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:55.883 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:55.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:55.957 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:55.971+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:56.008 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.008 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:56.058 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:56.072+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:56.143 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.144 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:26:56.159 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:56.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:56.265 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:56.279+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:56.284 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.284 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:26:56.380 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:26:56.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:26:56.462 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.462 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.462 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:56.476+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:56.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:26:56.563 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:56.578+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:56.644 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.644 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:26:56.665 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:56.679+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:56.712 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:26:56.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-06T09:26:56.765 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:56.779+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:56.822 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.822 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:26:56.866 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:56.880+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:56.966 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:56.966 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:56.967 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:56.981+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:56.975 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:26:56.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:26:57.068 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:57.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:57.155 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:57.202 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:57.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:26:57.204 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:57.218+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:57.307 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:57.320+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:57.375 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:57.376 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:57.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:26:57.406 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:57.421+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:57.508 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:57.522+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:57.512 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:57.513 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:57.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:26:57.608 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:57.623+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:57.653 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:57.653 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:57.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:26:57.710 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:57.724+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:57.795 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:57.795 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:57.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:26:57.846 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:57.860+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:57.911 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:57.925+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:57.973 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:57.973 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:57.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:26:58.023 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:58.037+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:58.113 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:26:58.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:26:58.153 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:58.153 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:58.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:26:58.213 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:58.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:58.292 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:58.292 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:58.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:26:58.343 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:58.357+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:58.416 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:58.430+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:58.470 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:58.470 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:58.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:26:58.522 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:58.536+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:58.618 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:58.632+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:58.651 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:58.651 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:58.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:26:58.719 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:58.734+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:58.819 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:58.833+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:58.829 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:58.829 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:58.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:26:58.920 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:26:58.935+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:26:58.978 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:58.978 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:58.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:26:59.022 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:59.036+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:59.122 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:59.136+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:59.157 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:59.157 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:59.166 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:26:59.223 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:26:59.238+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:26:59.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T09:26:59.293 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:59.293 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:59.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:26:59.324 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:59.338+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:59.426 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:26:59.440+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:26:59.473 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:59.473 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:59.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:26:59.527 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:59.541+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:59.611 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:59.611 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:59.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:26:59.628 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:59.642+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:26:59.729 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:26:59.743+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:26:59.753 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:59.753 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:59.762 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:26:59.831 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:26:59.844+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:26:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:26:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:26:59.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:26:59.932 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:26:59.946+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:00.033 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:00.046+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:00.038 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:00.038 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:00.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:00.133 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:00.147+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:00.176 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:00.176 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:00.185 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:00.234 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:00.248+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:00.335 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:00.348+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:00.353 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:00.353 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:00.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:00.448 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:00.461+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:00.531 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:00.531 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:00.536 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:00.549+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:00.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:00.637 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:00.651+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:00.708 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:00.708 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:00.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:00.738 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:00.752+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:00.850 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:00.864+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:00.862 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:00.863 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:00.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:00.940 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:00.954+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:01.041 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.041 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.041 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:01.055+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:01.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:01.143 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:01.157+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:01.180 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.180 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.194 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:01.244 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:01.258+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:01.322 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.323 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:01.345 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:01.358+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:01.445 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:01.459+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:01.458 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.458 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.467 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:01.546 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:01.560+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:01.595 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.596 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.608 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:01.652 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:01.666+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:01.738 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.738 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:01.749 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:01.762+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:01.848 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:01.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:01.916 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:01.916 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:01.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:01.949 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:01.963+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:02.050 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:02.064+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:02.051 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:02.051 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:02.060 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:02.151 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:02.165+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:02.234 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:02.234 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:02.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:02.286 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:02.300+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:02.353 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:02.367+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:02.422 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:02.423 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:02.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:02.454 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:02.467+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:02.554 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:02.568+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:02.602 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:02.602 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:02.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:02.656 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:02.670+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:02.751 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:02.751 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:02.756 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:02.769+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:02.761 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:02.857 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:02.871+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:02.928 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:02.928 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:02.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:02.981 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:02.995+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:03.060 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:03.073+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:03.064 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.064 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:03.160 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:03.174+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:03.298 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:03.312+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:03.366 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:03.376+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:03.381 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.382 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.390 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:03.463 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:03.477+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:03.558 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.559 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.564 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:03.578+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:03.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:03.665 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:03.679+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:03.696 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.696 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:03.765 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:03.780+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:03.835 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.835 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.844 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:03.867 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:03.880+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:03.968 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:03.982+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:03.977 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:03.978 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:03.986 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:04.068 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:04.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:04.114 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:04.115 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:04.124 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:04.208 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:04.222+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:04.270 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:04.284+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:04.291 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:04.292 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:04.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:04.371 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:04.385+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:04.471 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:04.471 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:04.471 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:04.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:04.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:04.573 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:04.587+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:04.620 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:04.620 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:04.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:04.674 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:04.688+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:04.764 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:04.764 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:04.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:04.774 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:04.788+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:04.876 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:04.890+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:04.944 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:04.944 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:04.955 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:04.998 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:05.012+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:05.077 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:05.091+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:05.084 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:05.084 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:05.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:05.178 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:05.192+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:05.262 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:05.262 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:05.271 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:05.315 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:05.329+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:05.380 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:05.394+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:05.398 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:05.398 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:05.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:05.494 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:05.508+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:05.581 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:05.581 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:05.581 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:05.595+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:05.590 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:05.682 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:05.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:05.758 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:05.758 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:05.767 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:05.811 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:05.825+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:05.884 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:05.898+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:05.893 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:05.894 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:05.903 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:05.985 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:05.999+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:06.071 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:06.071 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:06.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:06.086 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:06.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:06.187 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:06.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:06.210 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:06.210 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:06.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:06.288 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:06.302+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:06.390 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:06.403+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:06.391 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:06.392 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:06.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:06.490 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:06.504+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:06.540 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:06.540 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:06.550 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:06.591 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:06.606+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:06.692 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:06.706+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:06.720 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:06.720 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:06.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:06.793 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:06.807+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:06.894 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:06.908+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:06.904 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:06.904 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:06.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:06.995 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:07.009+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:07.083 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:07.084 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:07.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:07.096 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:07.110+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:07.197 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:07.211+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:07.223 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:07.223 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:07.233 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:07.298 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:07.312+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:07.363 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:07.363 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:07.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:07.401 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:07.412+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:07.500 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:07.513+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:07.502 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:07.502 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:07.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:07.601 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:07.615+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:07.641 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:07.642 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:07.651 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:07.735 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:07.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:07.750 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:07.803 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:07.817+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:07.823 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:07.823 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:07.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:07.842 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:07.842 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:07.904 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:07.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:07.942 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:07.942 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:08.001 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:08.001 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:08.005 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:08.019+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:08.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:08.031 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:08.106 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:08.120+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:08.125 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:08.125 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:08.127 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.1 to 1 (10000) 2023-12-06T09:27:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:27:08.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-06T09:27:08.197 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:08.198 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:08.208 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:08.228 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:08.242+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:08.237 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:08.307 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:08.321+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:08.327 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:08.327 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:08.359 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:08.359 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:08.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:08.408 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:08.422+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:08.417 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:08.417 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:08.509 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:08.523+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:08.515 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:08.516 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:08.516 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:08.516 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:08.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:08.610 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:08.624+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:08.699 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:08.700 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:08.700 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:08.700 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:08.700 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:08.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:08.710 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:08.724+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:08.811 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:08.825+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:08.839 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:08.839 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:08.839 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:08.839 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:27:08.839 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:08.849 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:08.933 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:08.947+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:08.977 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:27:08.991 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T09:27:08.991 INFO:tasks.mon_recovery:causing some monitor log activity 2023-12-06T09:27:08.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2023-12-06T09:27:09.013 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:09.027+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:09.022 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:09.022 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:09.022 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:09.022 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:09.031 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:09.114 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:09.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:09.209 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:09.209 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:09.210 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:09.210 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:09.210 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:09.215 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:09.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:09.219 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:09.316 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:09.330+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:09.414 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:09.414 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:09.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:09.414 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:09.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 312.49909526499999, 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:27:09.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:27:09.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:27:09.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.420 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 310.08138514799998, 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.421 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:27:09.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:27:09.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:27:09.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 310.07720070300002, 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.426 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:09.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:27:09.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:27:09.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:27:09.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.99166961700001, 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:27:09.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:27:09.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.67769300200001, 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:27:09.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:27:09.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:27:09.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:27:09.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:27:09.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.67617515799998, 2023-12-06T09:27:09.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:27:09.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:27:09.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.447 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.675509783, 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:09.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:27:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:27:09.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:27:09.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:27:09.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.67124961100001, 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:27:09.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:27:09.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:27:09.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:27:09.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.67061640200001, 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.458 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:27:09.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:27:09.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.463 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.6699893, 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:27:09.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:27:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:27:09.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:27:09.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.58172952199999, 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:27:09.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:27:09.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:27:09.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:27:09.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 309.580808542, 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:27:09.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:27:09.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:27:09.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.479 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 308.82843061099999, 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:27:09.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:27:09.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.484 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.95839563999999, 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.957222129, 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.490 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:27:09.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:27:09.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:27:09.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.495 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.94907015799998, 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.500 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.93878822200003, 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.93813034200002, 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.506 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.93745556300001, 2023-12-06T09:27:09.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 300.90509797200002, 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.517 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:09.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:27:09.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:27:09.522 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.522 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.522 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.522 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.522 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.522 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:09.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:09.524 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:09.537+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:09.624 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:09.638+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:09.658 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:09.658 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:09.658 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:09.658 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:09.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:09.726 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:09.740+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:09.827 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:09.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:09.835 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:09.835 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:09.836 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:09.836 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:09.836 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:09.844 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:09.928 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:09.942+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:09.977 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 159.342162495, 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:09.978 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:09.979 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.980 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.981 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 159.03694119400001, 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:09.982 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:09.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:09.984 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:09.985 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:09.988 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:10.029 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:10.042+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:10.129 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:10.143+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:10.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:27:10.165 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:10.165 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:10.165 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:10.165 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:10.167 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.2 to 1 (10000) 2023-12-06T09:27:10.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:10.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-06T09:27:10.230 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:10.244+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:10.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2023-12-06T09:27:10.340 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:10.354+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:10.378 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:10.378 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:10.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:10.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:10.432 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:10.446+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:10.533 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:10.546+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:10.599 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 295.63819996000001, 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:10.604 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:10.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:10.659 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:10.673+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:10.742 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:10.742 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:10.742 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:10.742 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:10.746 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:10.759+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:10.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:10.836 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:10.850+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:10.878 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:10.878 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:10.883 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:10.883 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:10.883 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:10.883 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:10.883 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:10.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:10.937 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:10.951+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:10.965 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:10.965 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:11.020 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:11.020 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:11.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:11.020 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:11.020 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:11.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:11.037 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:11.051+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:11.095 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:11.095 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:11.140 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:11.152+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:11.158 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:11.158 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:11.158 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:11.158 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:11.168 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:11.181 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:11.182 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:11.240 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:11.254+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:11.254 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:27:11.273 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:11.273 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:11.311 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.3 to 1 (10000) 2023-12-06T09:27:11.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-12-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:11.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:11.370 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:11.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:11.408 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:11.409 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:11.442 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:11.456+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:11.492 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:11.492 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:11.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:11.517 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:11.518 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:11.542 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:11.556+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:11.605 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:11.633 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:11.634 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:11.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:11.646 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:11.657+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:11.743 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:11.757+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:11.813 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:11.813 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:11.822 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:11.863 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:11.877+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:11.945 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:11.959+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:11.990 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:11.991 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:12.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:12.086 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:12.099+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:12.147 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:12.161+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:12.173 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:12.173 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:12.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:12.266 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:12.280+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:12.349 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:12.349 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:12.350 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:12.363+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:12.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:12.451 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:12.465+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.4 to 1 (10000) 2023-12-06T09:27:12.475 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2023-12-06T09:27:12.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-12-06T09:27:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:12.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:12.585 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:12.599+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:12.653 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:12.667+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:12.708 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:12.708 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:12.717 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:12.759 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:12.773+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:12.893 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:12.907+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:12.906 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:12.907 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:12.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:12.957 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:12.971+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:13.058 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:13.072+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:13.089 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:13.089 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:13.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:13.159 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:13.173+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:13.250 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:13.251 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:13.260 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:13.261 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:13.274+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:13.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:27:13.361 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:13.375+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:13.404 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:13.404 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:13.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:13.462 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:13.476+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:13.563 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:13.577+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:13.590 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:13.590 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:13.600 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:13.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:13.685 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:13.699+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:13.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2023-12-06T09:27:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:13.783 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:13.797+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:13.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:13.865 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:13.879+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:13.966 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:13.980+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:13.974 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:13.975 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:14.002 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:14.067 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:14.081+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:14.070 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:14.164 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:14.165 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:14.172 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:14.186+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:14.184 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:14.184 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:14.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:14.184 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:14.184 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 163.54775914499999, 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 163.242537844, 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:14.192 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.193 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:14.193 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.193 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.193 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:14.193 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:14.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:14.231 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:27:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:14.260 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:14.270 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:14.284+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:14.295 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.6 to 1 (10000) 2023-12-06T09:27:14.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-12-06T09:27:14.350 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:14.350 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:14.371 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:14.385+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:14.371 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:14.372 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:14.372 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:14.372 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:14.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:14.448 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:14.471 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:14.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:14.532 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:14.560 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:14.560 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:14.561 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:14.561 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:14.561 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:14.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:14.572 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:14.586+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:14.674 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:14.687+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:14.746 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:14.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:14.746 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:14.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 299.78440777999998, 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:14.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:14.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:14.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:14.775 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:14.789+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:14.876 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:14.890+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:14.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2023-12-06T09:27:14.894 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:14.894 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:14.894 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:14.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:14.986 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:15.000+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:15.071 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:15.072 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:15.072 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:15.072 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:15.072 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:15.078 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:15.092+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:15.080 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:15.179 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:15.193+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:15.252 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:15.252 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:15.252 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:15.252 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:15.252 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:15.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:15.280 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:15.294+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:15.381 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:15.395+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:15.390 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:15.390 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:15.390 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:15.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:15.482 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:15.496+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:15.568 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:15.568 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:15.568 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:15.569 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:15.569 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:15.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:15.583 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:15.597+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:15.698 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:15.711+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 317.04676200400002, 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:27:15.753 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.754 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.755 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:27:15.756 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 316.35230813499999, 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:27:15.758 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.759 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:27:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:27:15.761 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.762 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.763 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.90941565899999, 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.764 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:27:15.765 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:27:15.766 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.90058937800001, 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:27:15.768 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.770 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.770 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:15.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.772 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:27:15.773 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.774 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.89996770599998, 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.775 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.777 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:27:15.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.780 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.89202122500001, 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.784 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.89154720699997, 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:15.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.89109285199999, 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.789 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:27:15.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:27:15.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:27:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.50710060099999, 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.793 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:27:15.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:27:15.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:27:15.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.797 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.121951196, 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.800 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.800 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:27:15.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:27:15.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.801 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.109675885, 2023-12-06T09:27:15.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:15.807 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 315.10819030699997, 2023-12-06T09:27:15.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:27:15.809 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.809 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.809 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.813 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.20165314899998, 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.814 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:27:15.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:27:15.817 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.196520846, 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:15.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:27:15.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:27:15.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:27:15.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.19569481600001, 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.19312535500001, 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.826 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:27:15.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:27:15.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.830 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.19237988499998, 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:27:15.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:27:15.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:27:15.834 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.19157381399998, 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:15.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:27:15.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:27:15.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:27:15.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 307.180469362, 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 158.40713722500001, 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:27:15.843 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:27:15.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.846 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.847 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:15.847 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:15.847 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:15.849 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2023-12-06T09:27:15.849 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:15.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:15.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:15.938 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.7 to 1 (10000) 2023-12-06T09:27:15.950 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:15.964+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:15.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-06T09:27:16.025 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:16.025 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:16.025 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:16.025 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:16.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:16.076 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:16.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:16.151 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:16.165+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:16.204 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:16.204 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:16.204 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:16.204 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:16.204 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:16.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:16.248 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:27:16.256 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:16.269+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:16.354 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:16.368+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 1.18346391, 2023-12-06T09:27:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010005948000000001, 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 11 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.405 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221310+0000", 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6829000000000001e-05 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221505+0000", 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019430599999999999 2023-12-06T09:27:16.406 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221716+0000", 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021168399999999999 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.224250+0000", 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025338729999999999 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.229715+0000", 2023-12-06T09:27:16.407 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054648589999999999 2023-12-06T09:27:16.408 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:16.408 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:16.408 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:16.408 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.231300+0000", 2023-12-06T09:27:16.408 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001584397 2023-12-06T09:27:16.408 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:16.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:16.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:16.409 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:16.409 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:16.409 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:16.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:16.457 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:16.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:16.542 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:16.542 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:16.543 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:16.543 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:16.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:16.593 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:16.607+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:16.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:16.693 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:16.705+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:16.729 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:16.729 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:16.729 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:16.729 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:16.729 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:16.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:16.782 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:16.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:16.858 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:16.872+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:16.870 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:16.870 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:16.870 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:16.870 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:16.870 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:16.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:16.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2023-12-06T09:27:16.910 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.0 to 1 (10000) 2023-12-06T09:27:16.918 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:16.918 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:16.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-12-06T09:27:16.970 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:16.984+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:17.036 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:17.036 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:17.046 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:17.060 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:17.074+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:17.067 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:17.067 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:17.161 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:17.175+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:17.162 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:17.189 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:17.189 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:17.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:17.284 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:17.285 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:17.285 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:17.299+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:17.363 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:17.377+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:17.372 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:17.373 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:17.375 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:17.375 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:17.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:17.462 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:17.462 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:17.464 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:17.478+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:17.548 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:17.554 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:17.554 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:17.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:17.605 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:17.619+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:17.632 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:17.633 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:17.686 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:17.700+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:17.734 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:17.735 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:17.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:17.767 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:17.781+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:17.820 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:50.134 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:17.821 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:55.799 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:17.825 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:55.804 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:17.838 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:50.153 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:17.850 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:55.829 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:17.866 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:00:50.180 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:17.868 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:17.882+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:17.918 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:17.918 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:17.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:17.971 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:17.984+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:18.070 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:18.084+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:18.103 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:18.104 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:18.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:18.125 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.5 to 1 (10000) 2023-12-06T09:27:18.140 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-06T09:27:18.140 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:27:18.171 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:18.185+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:18.220 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:27:18.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2023-12-06T09:27:18.272 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:18.286+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:18.276 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:18.276 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:18.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:18.373 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:18.387+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:18.462 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:18.462 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:18.472 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:18.474 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:18.488+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:18.575 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:18.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:18.642 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:18.643 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:18.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:18.696 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:18.710+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:18.777 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:18.791+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:18.795 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:27:18.796 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:27:18.810 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2023-12-06T09:27:18.166783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222820,"kb_used_data":5672,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754751900,"statfs":{"total":773094113280,"available":772865945600,"internally_reserved":0,"allocated":5808128,"data_stored":2937923,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.192964"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:13.975979+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:27:13.975979+0000","last_peered":"2023-12-06T09:27:13.975979+0000","last_clean":"2023-12-06T09:27:13.975979+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:27:13.975979+0000","last_undegraded":"2023-12-06T09:27:13.975979+0000","last_fullsized":"2023-12-06T09:27:13.975979+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":84,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.452503+0000","last_change":"2023-12-06T09:23:50.752271+0000","last_active":"2023-12-06T09:27:12.452503+0000","last_peered":"2023-12-06T09:27:12.452503+0000","last_clean":"2023-12-06T09:27:12.452503+0000","last_became_active":"2023-12-06T09:23:50.743293+0000","last_became_peered":"2023-12-06T09:23:50.743293+0000","last_unstale":"2023-12-06T09:27:12.452503+0000","last_undegraded":"2023-12-06T09:27:12.452503+0000","last_fullsized":"2023-12-06T09:27:12.452503+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:27:50.211606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-06T09:27:15.697415+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:27:15.697415+0000","last_peered":"2023-12-06T09:27:15.697415+0000","last_clean":"2023-12-06T09:27:15.697415+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:27:15.697415+0000","last_undegraded":"2023-12-06T09:27:15.697415+0000","last_fullsized":"2023-12-06T09:27:15.697415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:13.973994+0000","last_change":"2023-12-06T09:24:37.205991+0000","last_active":"2023-12-06T09:27:13.973994+0000","last_peered":"2023-12-06T09:27:13.973994+0000","last_clean":"2023-12-06T09:27:13.973994+0000","last_became_active":"2023-12-06T09:24:37.200368+0000","last_became_peered":"2023-12-06T09:24:37.200368+0000","last_unstale":"2023-12-06T09:27:13.973994+0000","last_undegraded":"2023-12-06T09:27:13.973994+0000","last_fullsized":"2023-12-06T09:27:13.973994+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:57:43.491211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.448736+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:27:12.448736+0000","last_peered":"2023-12-06T09:27:12.448736+0000","last_clean":"2023-12-06T09:27:12.448736+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:27:12.448736+0000","last_undegraded":"2023-12-06T09:27:12.448736+0000","last_fullsized":"2023-12-06T09:27:12.448736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.490909+0000","last_change":"2023-12-06T09:23:45.621641+0000","last_active":"2023-12-06T09:27:12.490909+0000","last_peered":"2023-12-06T09:27:12.490909+0000","last_clean":"2023-12-06T09:27:12.490909+0000","last_became_active":"2023-12-06T09:23:45.615923+0000","last_became_peered":"2023-12-06T09:23:45.615923+0000","last_unstale":"2023-12-06T09:27:12.490909+0000","last_undegraded":"2023-12-06T09:27:12.490909+0000","last_fullsized":"2023-12-06T09:27:12.490909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:06:07.386710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":87,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-06T09:27:15.686757+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:27:15.686757+0000","last_peered":"2023-12-06T09:27:15.686757+0000","last_clean":"2023-12-06T09:27:15.686757+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:27:15.686757+0000","last_undegraded":"2023-12-06T09:27:15.686757+0000","last_fullsized":"2023-12-06T09:27:15.686757+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.493466+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:27:12.493466+0000","last_peered":"2023-12-06T09:27:12.493466+0000","last_clean":"2023-12-06T09:27:12.493466+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:27:12.493466+0000","last_undegraded":"2023-12-06T09:27:12.493466+0000","last_fullsized":"2023-12-06T09:27:12.493466+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":201,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:13.972479+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:27:13.972479+0000","last_peered":"2023-12-06T09:27:13.972479+0000","last_clean":"2023-12-06T09:27:13.972479+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:27:13.972479+0000","last_undegraded":"2023-12-06T09:27:13.972479+0000","last_fullsized":"2023-12-06T09:27:13.972479+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509507,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":737280,"data_stored":274958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":9.5999999999999996,"5min":10.960000000000001,"15min":14.996},"min":{"1min":4.4269999999999996,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":20.725999999999999,"5min":49.691000000000003,"15min":49.691000000000003},"last":17.021000000000001},{"interface":"front","average":{"1min":8.8949999999999996,"5min":10.445,"15min":14.975},"min":{"1min":5.8170000000000002,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":16.494,"5min":51.530000000000001,"15min":51.530000000000001},"last":7.6900000000000004}]},{"osd":1,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.195,"5min":10.938000000000001,"15min":12.022},"min":{"1min":5.7530000000000001,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":40.991,"5min":40.991,"15min":40.991},"last":8.4499999999999993},{"interface":"front","average":{"1min":15.298,"5min":13.292,"15min":14.442},"min":{"1min":8.1880000000000006,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":35.850000000000001,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.501999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.053000000000001,"5min":12.885,"15min":14.698},"min":{"1min":7.3819999999999997,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":18.181999999999999,"5min":42.523000000000003,"15min":42.523000000000003},"last":14.212},{"interface":"front","average":{"1min":14.087,"5min":15.260999999999999,"15min":17.285},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":21.725999999999999,"5min":39.066000000000003,"15min":39.066000000000003},"last":6.7569999999999997}]},{"osd":4,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.579000000000001,"5min":13.173,"15min":14.601000000000001},"min":{"1min":9.1319999999999997,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":34.061,"5min":34.061,"15min":34.061},"last":22.126999999999999},{"interface":"front","average":{"1min":19.207999999999998,"5min":15.503,"15min":15.571},"min":{"1min":11.127000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":32.462000000000003,"5min":32.462000000000003,"15min":32.462000000000003},"last":21.379999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.001999999999999,"5min":16.791,"15min":24.204999999999998},"min":{"1min":8.8490000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":24.170999999999999,"5min":190.821,"15min":190.821},"last":20.484999999999999},{"interface":"front","average":{"1min":17.789999999999999,"5min":18.347000000000001,"15min":24.542999999999999},"min":{"1min":6.1950000000000003,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":44.625999999999998,"5min":191.86199999999999,"15min":191.86199999999999},"last":25.943999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":20.004000000000001,"5min":18.93,"15min":24.388999999999999},"min":{"1min":11.297000000000001,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":33.362000000000002,"5min":166.45400000000001,"15min":166.45400000000001},"last":18.919},{"interface":"front","average":{"1min":18.824999999999999,"5min":19.602,"15min":24.550000000000001},"min":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.606000000000002,"5min":117.58499999999999,"15min":117.58499999999999},"last":25.146999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":23.681000000000001,"5min":23.681000000000001,"15min":23.681000000000001},"min":{"1min":12.381,"5min":12.381,"15min":12.381},"max":{"1min":52.109000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":22.834},{"interface":"front","average":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"min":{"1min":11.01,"5min":11.01,"15min":11.01},"max":{"1min":56.914000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":28.093}]}]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.0969999999999995,"15min":8.1890000000000001},"min":{"1min":6.0819999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":25.431000000000001,"5min":38.481000000000002,"15min":38.481000000000002},"last":8.1489999999999991},{"interface":"front","average":{"1min":10.859,"5min":11.725,"15min":12.146000000000001},"min":{"1min":5.7930000000000001,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":26.995000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":6.9189999999999996}]},{"osd":1,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":13.384,"15min":14.712},"min":{"1min":9.0299999999999994,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":26.780000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":10.257},{"interface":"front","average":{"1min":16.044,"5min":16.731999999999999,"15min":18.222000000000001},"min":{"1min":9.6699999999999999,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":28.262,"5min":54.338000000000001,"15min":54.338000000000001},"last":11.132}]},{"osd":3,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":11.717000000000001,"15min":11.206},"min":{"1min":7.7359999999999998,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":21.158000000000001,"5min":27.295999999999999,"15min":27.295999999999999},"last":32.860999999999997},{"interface":"front","average":{"1min":11.641,"5min":11.231,"15min":12.781000000000001},"min":{"1min":6.468,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":19.015999999999998,"5min":28.669,"15min":28.669},"last":11.763999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.824,"5min":14.007,"15min":14.811},"min":{"1min":7.2889999999999997,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":20.074999999999999,"5min":54.32,"15min":54.32},"last":19.866},{"interface":"front","average":{"1min":15.898,"5min":16.734000000000002,"15min":17.497},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":26.274000000000001,"5min":51.950000000000003,"15min":51.950000000000003},"last":12.297000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.038,"15min":15.157},"min":{"1min":7.0330000000000004,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":48.563000000000002,"5min":56.491,"15min":56.491},"last":21.747},{"interface":"front","average":{"1min":16.576000000000001,"5min":16.488,"15min":19.004000000000001},"min":{"1min":9.5960000000000001,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":47.328000000000003,"5min":75.540999999999997,"15min":75.540999999999997},"last":22.581}]},{"osd":6,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.277999999999999,"15min":18.053000000000001},"min":{"1min":10.872999999999999,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":28.962,"5min":30.213999999999999,"15min":30.213999999999999},"last":13.906000000000001},{"interface":"front","average":{"1min":14.414,"5min":13.6,"15min":14.318},"min":{"1min":8.2479999999999993,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":24.893999999999998,"5min":26.312999999999999,"15min":26.312999999999999},"last":24.859999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:58 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":17.641999999999999,"15min":17.641999999999999},"min":{"1min":12.304,"5min":12.304,"15min":12.304},"max":{"1min":30.283000000000001,"5min":30.283000000000001,"15min":30.283000000000001},"last":20.814},{"interface":"front","average":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":19.783000000000001},"min":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":31.402999999999999,"5min":31.402999999999999,"15min":31.402999999999999},"last":23.962}]}]},{"osd":0,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":270541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.576000000000001,"5min":11.565,"15min":11.337999999999999},"min":{"1min":4.8289999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":52.207000000000001,"5min":65.808000000000007,"15min":65.808000000000007},"last":8.4589999999999996},{"interface":"front","average":{"1min":12.784000000000001,"5min":12.163,"15min":11.606999999999999},"min":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":53.106999999999999,"5min":53.106999999999999,"15min":53.106999999999999},"last":13.702}]},{"osd":2,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.065,"15min":13.079000000000001},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":41.301000000000002,"5min":49.652999999999999,"15min":49.652999999999999},"last":17.263000000000002},{"interface":"front","average":{"1min":12.539,"5min":13.265000000000001,"15min":14.048},"min":{"1min":7.9960000000000004,"5min":6.798,"15min":6.798},"max":{"1min":21.843,"5min":47.484000000000002,"15min":47.484000000000002},"last":9.2100000000000009}]},{"osd":3,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.029,"5min":13.706,"15min":20.071999999999999},"min":{"1min":6.7320000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":32.063000000000002,"5min":205.096,"15min":205.096},"last":11.638999999999999},{"interface":"front","average":{"1min":18.690000000000001,"5min":17.489000000000001,"15min":22.172000000000001},"min":{"1min":8.5660000000000007,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":45.463000000000001,"5min":182.74000000000001,"15min":182.74000000000001},"last":16.628}]},{"osd":4,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.943,"5min":13.228,"15min":13.978},"min":{"1min":8.0660000000000007,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":29.585999999999999,"5min":45.112000000000002,"15min":45.112000000000002},"last":31.451000000000001},{"interface":"front","average":{"1min":17.719000000000001,"5min":16.870999999999999,"15min":16.803000000000001},"min":{"1min":7.1470000000000002,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":34.533000000000001,"5min":52.655999999999999,"15min":52.655999999999999},"last":30.16}]},{"osd":5,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":15.237,"5min":15.542,"15min":15.226000000000001},"min":{"1min":9.2129999999999992,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":25.379999999999999,"5min":43.088000000000001,"15min":43.088000000000001},"last":10.427},{"interface":"front","average":{"1min":16.452999999999999,"5min":16.266999999999999,"15min":16.77},"min":{"1min":9.7230000000000008,"5min":8.266,"15min":8.266},"max":{"1min":31.550000000000001,"5min":45.722000000000001,"15min":45.722000000000001},"last":7.7709999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":21.395,"5min":19.311,"15min":20.494},"min":{"1min":9.2309999999999999,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":44.292999999999999,"5min":78.338999999999999,"15min":78.338999999999999},"last":27.809000000000001},{"interface":"front","average":{"1min":18.216000000000001,"5min":17.864999999999998,"15min":20.274000000000001},"min":{"1min":10.634,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":42.417999999999999,"5min":80.850999999999999,"15min":80.850999999999999},"last":15.49}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":18.908999999999999,"5min":18.908999999999999,"15min":18.908999999999999},"min":{"1min":8.2469999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":61.445999999999998,"5min":61.445999999999998,"15min":61.445999999999998},"last":12.217000000000001},{"interface":"front","average":{"1min":20.067,"5min":20.067,"15min":20.067},"min":{"1min":11.833,"5min":11.833,"15min":11.833},"max":{"1min":63.518000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":14.566000000000001}]}]},{"osd":7,"up_from":60,"seq":257698037778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28692,"kb_used_data":708,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343148,"statfs":{"total":96636764160,"available":96607383552,"internally_reserved":0,"allocated":724992,"data_stored":270541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":11.984999999999999,"5min":11.984999999999999,"15min":11.984999999999999},"min":{"1min":4.452,"5min":4.452,"15min":4.452},"max":{"1min":55.622,"5min":55.622,"15min":55.622},"last":6.984},{"interface":"front","average":{"1min":12.089,"5min":12.089,"15min":12.089},"min":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":31.178000000000001,"5min":31.178000000000001,"15min":31.178000000000001},"last":25.652999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":48.959000000000003,"5min":48.959000000000003,"15min":48.959000000000003},"last":28.062999999999999},{"interface":"front","average":{"1min":13.994999999999999,"5min":13.994999999999999,"15min":13.994999999999999},"min":{"1min":8.0380000000000003,"5min":8.0380000000000003,"15min":8.0380000000000003},"max":{"1min":38.232999999999997,"5min":38.232999999999997,"15min":38.232999999999997},"last":13.35}]},{"osd":2,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":12.195,"5min":12.195,"15min":12.195},"min":{"1min":5.952,"5min":5.952,"15min":5.952},"max":{"1min":20.297999999999998,"5min":20.297999999999998,"15min":20.297999999999998},"last":22.370000000000001},{"interface":"front","average":{"1min":15.340999999999999,"5min":15.340999999999999,"15min":15.340999999999999},"min":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":48.240000000000002,"5min":48.240000000000002,"15min":48.240000000000002},"last":29.916}]},{"osd":3,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":13.568,"15min":13.568},"min":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"max":{"1min":54.603999999999999,"5min":54.603999999999999,"15min":54.603999999999999},"last":20.783000000000001},{"interface":"front","average":{"1min":21.209,"5min":21.209,"15min":21.209},"min":{"1min":12.032999999999999,"5min":12.032999999999999,"15min":12.032999999999999},"max":{"1min":52.719000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":23.792000000000002}]},{"osd":4,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.044,"5min":15.044,"15min":15.044},"min":{"1min":7.2329999999999997,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":32.390999999999998,"5min":32.390999999999998,"15min":32.390999999999998},"last":30.542000000000002},{"interface":"front","average":{"1min":13.667999999999999,"5min":13.667999999999999,"15min":13.667999999999999},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":23.030999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.632999999999999,"5min":19.632999999999999,"15min":19.632999999999999},"min":{"1min":12.234,"5min":12.234,"15min":12.234},"max":{"1min":52.197000000000003,"5min":52.197000000000003,"15min":52.197000000000003},"last":26.657},{"interface":"front","average":{"1min":17.359999999999999,"5min":17.359999999999999,"15min":17.359999999999999},"min":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":38.877000000000002,"5min":38.877000000000002,"15min":38.877000000000002},"last":31.157}]},{"osd":6,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.231000000000002,"5min":19.231000000000002,"15min":19.231000000000002},"min":{"1min":12.494999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":39.508000000000003,"5min":39.508000000000003,"15min":39.508000000000003},"last":32.341000000000001},{"interface":"front","average":{"1min":14.912000000000001,"5min":14.912000000000001,"15min":14.912000000000001},"min":{"1min":8.6530000000000005,"5min":8.6530000000000005,"15min":8.6530000000000005},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":21.428000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":8.9559999999999995,"5min":10.226000000000001,"15min":11.521000000000001},"min":{"1min":4.3490000000000002,"5min":3.484,"15min":3.484},"max":{"1min":16.178999999999998,"5min":45.506,"15min":45.506},"last":11.682},{"interface":"front","average":{"1min":9.0839999999999996,"5min":10.619999999999999,"15min":12.609999999999999},"min":{"1min":5.1120000000000001,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":16.661000000000001,"5min":46.450000000000003,"15min":46.450000000000003},"last":8.1010000000000009}]},{"osd":1,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":11.772,"5min":13.038,"15min":15.023999999999999},"min":{"1min":7.4740000000000002,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":20.497,"5min":55.398000000000003,"15min":55.398000000000003},"last":10.119999999999999},{"interface":"front","average":{"1min":13.172000000000001,"5min":13.750999999999999,"15min":14.923},"min":{"1min":8.9809999999999999,"5min":6.569,"15min":6.569},"max":{"1min":22.056999999999999,"5min":41.811999999999998,"15min":41.811999999999998},"last":17.503}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":15.488,"15min":20.422000000000001},"min":{"1min":11.131,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":23.66,"5min":208.93799999999999,"15min":208.93799999999999},"last":18.369},{"interface":"front","average":{"1min":13.587,"5min":16.247,"15min":24.045999999999999},"min":{"1min":7.3310000000000004,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":24.765000000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":13.029999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":17.373999999999999,"5min":15.544,"15min":17.196999999999999},"min":{"1min":10.898999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":25.806999999999999,"5min":43.276000000000003,"15min":43.276000000000003},"last":24.34},{"interface":"front","average":{"1min":9.9079999999999995,"5min":10.916,"15min":14.124000000000001},"min":{"1min":6.4320000000000004,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":17.137,"5min":41.493000000000002,"15min":41.493000000000002},"last":10.882999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":16.132000000000001,"5min":16.742999999999999,"15min":22.754999999999999},"min":{"1min":9.4000000000000004,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":27.443999999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":15.029999999999999},{"interface":"front","average":{"1min":18.768000000000001,"5min":18.789000000000001,"15min":25.478000000000002},"min":{"1min":8.4570000000000007,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":29.385000000000002,"5min":111.874,"15min":111.874},"last":25.707000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":12.522,"5min":12.273,"15min":16.116},"min":{"1min":7.3929999999999998,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":22.010000000000002,"5min":39.886000000000003,"15min":39.886000000000003},"last":12.359999999999999},{"interface":"front","average":{"1min":17.369,"5min":17.016999999999999,"15min":21.588999999999999},"min":{"1min":11.013999999999999,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":30.199999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":16.481000000000002}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":20.678999999999998,"5min":20.678999999999998,"15min":20.678999999999998},"min":{"1min":13.064,"5min":13.064,"15min":13.064},"max":{"1min":31.757999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":27.405000000000001},{"interface":"front","average":{"1min":15.526999999999999,"5min":15.526999999999999,"15min":15.526999999999999},"min":{"1min":9.6890000000000001,"5min":9.6890000000000001,"15min":9.6890000000000001},"max":{"1min":28.809000000000001,"5min":28.809000000000001,"15min":28.809000000000001},"last":29.145}]}]},{"osd":5,"up_from":14,"seq":60129542210,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":663964,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":9.0600000000000005,"5min":10.461,"15min":13.067},"min":{"1min":4.5670000000000002,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":18.443000000000001,"5min":45.491,"15min":45.491},"last":38.012},{"interface":"front","average":{"1min":8.7940000000000005,"5min":10.765000000000001,"15min":12.897},"min":{"1min":4.4349999999999996,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":14.475,"5min":49.991999999999997,"15min":49.991999999999997},"last":36.920999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":11.146000000000001,"5min":13.614000000000001,"15min":21.236000000000001},"min":{"1min":5.6900000000000004,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":18.754000000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":16.754000000000001},{"interface":"front","average":{"1min":10.789,"5min":17.363,"15min":29.614999999999998},"min":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":17.491,"5min":344.79599999999999,"15min":344.79599999999999},"last":10.994999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.129,"5min":12.884,"15min":13.379},"min":{"1min":9.1910000000000007,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":24.969999999999999,"5min":53.235999999999997,"15min":53.235999999999997},"last":7.3490000000000002},{"interface":"front","average":{"1min":14.458,"5min":13.545999999999999,"15min":15.108000000000001},"min":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":26.155000000000001,"5min":54.747,"15min":54.747},"last":15.968}]},{"osd":3,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.762,"5min":14.917,"15min":13.975},"min":{"1min":9.6069999999999993,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":24.18,"5min":55.310000000000002,"15min":55.310000000000002},"last":15.468999999999999},{"interface":"front","average":{"1min":11.239000000000001,"5min":12.801,"15min":16.099},"min":{"1min":6.9889999999999999,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":19.765000000000001,"5min":51.183,"15min":51.183},"last":13.975}]},{"osd":4,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":12.808,"5min":14.959,"15min":19.792999999999999},"min":{"1min":9.0570000000000004,"5min":6.085,"15min":6.085},"max":{"1min":21.306999999999999,"5min":281.608,"15min":281.608},"last":8.375},{"interface":"front","average":{"1min":16.542999999999999,"5min":16.850999999999999,"15min":21.760999999999999},"min":{"1min":11.683999999999999,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":23.295999999999999,"5min":225.83600000000001,"15min":225.83600000000001},"last":11.868}]},{"osd":6,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":16.638999999999999,"5min":16.809000000000001,"15min":20.645},"min":{"1min":11.105,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":22.853000000000002,"5min":198.667,"15min":198.667},"last":12.465999999999999},{"interface":"front","average":{"1min":16.501000000000001,"5min":17.471,"15min":22.067},"min":{"1min":10.128,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":24.225000000000001,"5min":208.446,"15min":208.446},"last":17.536000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":14.349,"5min":14.349,"15min":14.349},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":9.9860000000000007},{"interface":"front","average":{"1min":18.501000000000001,"5min":18.501000000000001,"15min":18.501000000000001},"min":{"1min":12.881,"5min":12.881,"15min":12.881},"max":{"1min":25.683,"5min":25.683,"15min":25.683},"last":20.109000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542210,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":737280,"data_stored":659547,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.8759999999999994,"5min":8.9649999999999999,"15min":10.57},"min":{"1min":5.6459999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":26.015000000000001,"5min":40.225999999999999,"15min":40.225999999999999},"last":16.841999999999999},{"interface":"front","average":{"1min":11.209,"5min":10.295999999999999,"15min":11.488},"min":{"1min":5.8639999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":20.613,"5min":30.405000000000001,"15min":30.405000000000001},"last":5.3319999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":10.425000000000001,"5min":13.679,"15min":11.686},"min":{"1min":7.0549999999999997,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":21.376000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":17.466000000000001},{"interface":"front","average":{"1min":15.489000000000001,"5min":17.225999999999999,"15min":16.045999999999999},"min":{"1min":10.004,"5min":6.415,"15min":6.415},"max":{"1min":30.391999999999999,"5min":164.911,"15min":164.911},"last":22.452999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.678000000000001,"5min":14.433999999999999,"15min":19.654},"min":{"1min":5.359,"5min":5.359,"15min":5.359},"max":{"1min":20.265999999999998,"5min":122.762,"15min":122.762},"last":8.1989999999999998},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.653,"15min":19.021000000000001},"min":{"1min":7.1059999999999999,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":22.109000000000002,"5min":129.99100000000001,"15min":129.99100000000001},"last":23.276}]},{"osd":4,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.468999999999999,"5min":18.315000000000001,"15min":27.951000000000001},"min":{"1min":9.1140000000000008,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":49.097000000000001,"5min":359.46300000000002,"15min":359.46300000000002},"last":9.1699999999999999},{"interface":"front","average":{"1min":15.288,"5min":17.658999999999999,"15min":26.722999999999999},"min":{"1min":6.4539999999999997,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":47.192,"5min":314.47699999999998,"15min":314.47699999999998},"last":19.954999999999998}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":19.167999999999999,"5min":21.059999999999999,"15min":25.846},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":81.923000000000002,"5min":152.97399999999999,"15min":152.97399999999999},"last":20.901},{"interface":"front","average":{"1min":16.448,"5min":18.698,"15min":24.085000000000001},"min":{"1min":10.108000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.151000000000003,"5min":154.53100000000001,"15min":154.53100000000001},"last":18.18}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.266999999999999,"5min":15.220000000000001,"15min":17.032},"min":{"1min":8.218,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":20.634,"5min":43.045000000000002,"15min":43.045000000000002},"last":24.062000000000001},{"interface":"front","average":{"1min":16.388000000000002,"5min":17.977,"15min":20.286999999999999},"min":{"1min":11.009,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":27.966999999999999,"5min":41.356000000000002,"15min":41.356000000000002},"last":15.733000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":19.687999999999999,"15min":19.687999999999999},"min":{"1min":11.547000000000001,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":32.191000000000003,"5min":32.191000000000003,"15min":32.191000000000003},"last":24.834},{"interface":"front","average":{"1min":19.468,"5min":19.468,"15min":19.468},"min":{"1min":13.093,"5min":13.093,"15min":13.093},"max":{"1min":36.454000000000001,"5min":36.454000000000001,"15min":36.454000000000001},"last":26.558}]}]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5269999999999992,"5min":11.606,"15min":13.595000000000001},"min":{"1min":4.9740000000000002,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":23.562000000000001,"5min":137.13999999999999,"15min":137.13999999999999},"last":7.1269999999999998},{"interface":"front","average":{"1min":9.0630000000000006,"5min":12.724,"15min":17.661999999999999},"min":{"1min":4.1959999999999997,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":23.231000000000002,"5min":199.268,"15min":199.268},"last":13.186999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.608000000000001,"15min":18.324000000000002},"min":{"1min":5.2779999999999996,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":25.949999999999999,"5min":92.817999999999998,"15min":92.817999999999998},"last":10.518000000000001},{"interface":"front","average":{"1min":11.914,"5min":14.56,"15min":19.475999999999999},"min":{"1min":7.8040000000000003,"5min":5.556,"15min":5.556},"max":{"1min":17.295000000000002,"5min":95.665999999999997,"15min":95.665999999999997},"last":5.6580000000000004}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":13.292999999999999,"5min":17.664000000000001,"15min":23.103999999999999},"min":{"1min":7.4470000000000001,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":23.303999999999998,"5min":208.06,"15min":208.06},"last":14.835000000000001},{"interface":"front","average":{"1min":13.791,"5min":16.366,"15min":18.635999999999999},"min":{"1min":8.7569999999999997,"5min":6.915,"15min":6.915},"max":{"1min":21.555,"5min":147.18199999999999,"15min":147.18199999999999},"last":24.934999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":13.999000000000001,"15min":18.155000000000001},"min":{"1min":6.2610000000000001,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":28.175999999999998,"5min":121.627,"15min":121.627},"last":12.118},{"interface":"front","average":{"1min":15.657999999999999,"5min":16.553999999999998,"15min":18.242000000000001},"min":{"1min":10.692,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":25.760999999999999,"5min":74.602000000000004,"15min":74.602000000000004},"last":11.311}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":16.504000000000001,"5min":16.984000000000002,"15min":18.885000000000002},"min":{"1min":8.3930000000000007,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":24.431000000000001,"5min":71.759,"15min":71.759},"last":16.286999999999999},{"interface":"front","average":{"1min":13.960000000000001,"5min":14.284000000000001,"15min":16.748000000000001},"min":{"1min":6.6399999999999997,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":29.401,"5min":70.882999999999996,"15min":70.882999999999996},"last":14.121}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.581,"5min":16.417999999999999,"15min":19.870000000000001},"min":{"1min":8.9700000000000006,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":22.050999999999998,"5min":80.668999999999997,"15min":80.668999999999997},"last":25.928000000000001},{"interface":"front","average":{"1min":17.324999999999999,"5min":19.062999999999999,"15min":21.738},"min":{"1min":12.243,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":25.762,"5min":72.796999999999997,"15min":72.796999999999997},"last":18.181000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":17.497,"5min":17.497,"15min":17.497},"min":{"1min":11.442,"5min":11.442,"15min":11.442},"max":{"1min":29.594000000000001,"5min":29.594000000000001,"15min":29.594000000000001},"last":26.902999999999999},{"interface":"front","average":{"1min":18.670999999999999,"5min":18.670999999999999,"15min":18.670999999999999},"min":{"1min":11.663,"5min":11.663,"15min":11.663},"max":{"1min":27.414000000000001,"5min":27.414000000000001,"15min":27.414000000000001},"last":17.550999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:27:18.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:27:18.815 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:18.815 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:18.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:18.913 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:18.927+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:18.990 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:19.004+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:19.005 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:19.005 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:19.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:19.080 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:19.094+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:19.143 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:19.143 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:19.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:19.181 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:19.195+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:19.282 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:19.296+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:19.288 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:19.288 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:19.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:19.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2023-12-06T09:27:19.383 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:19.397+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:19.388 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:27:19.389 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:27:19.402 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2023-12-06T09:27:18.166783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222820,"kb_used_data":5672,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754751900,"statfs":{"total":773094113280,"available":772865945600,"internally_reserved":0,"allocated":5808128,"data_stored":2937923,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.192964"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:13.975979+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:27:13.975979+0000","last_peered":"2023-12-06T09:27:13.975979+0000","last_clean":"2023-12-06T09:27:13.975979+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:27:13.975979+0000","last_undegraded":"2023-12-06T09:27:13.975979+0000","last_fullsized":"2023-12-06T09:27:13.975979+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":84,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.452503+0000","last_change":"2023-12-06T09:23:50.752271+0000","last_active":"2023-12-06T09:27:12.452503+0000","last_peered":"2023-12-06T09:27:12.452503+0000","last_clean":"2023-12-06T09:27:12.452503+0000","last_became_active":"2023-12-06T09:23:50.743293+0000","last_became_peered":"2023-12-06T09:23:50.743293+0000","last_unstale":"2023-12-06T09:27:12.452503+0000","last_undegraded":"2023-12-06T09:27:12.452503+0000","last_fullsized":"2023-12-06T09:27:12.452503+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:27:50.211606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-06T09:27:15.697415+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:27:15.697415+0000","last_peered":"2023-12-06T09:27:15.697415+0000","last_clean":"2023-12-06T09:27:15.697415+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:27:15.697415+0000","last_undegraded":"2023-12-06T09:27:15.697415+0000","last_fullsized":"2023-12-06T09:27:15.697415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:13.973994+0000","last_change":"2023-12-06T09:24:37.205991+0000","last_active":"2023-12-06T09:27:13.973994+0000","last_peered":"2023-12-06T09:27:13.973994+0000","last_clean":"2023-12-06T09:27:13.973994+0000","last_became_active":"2023-12-06T09:24:37.200368+0000","last_became_peered":"2023-12-06T09:24:37.200368+0000","last_unstale":"2023-12-06T09:27:13.973994+0000","last_undegraded":"2023-12-06T09:27:13.973994+0000","last_fullsized":"2023-12-06T09:27:13.973994+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:57:43.491211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.448736+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:27:12.448736+0000","last_peered":"2023-12-06T09:27:12.448736+0000","last_clean":"2023-12-06T09:27:12.448736+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:27:12.448736+0000","last_undegraded":"2023-12-06T09:27:12.448736+0000","last_fullsized":"2023-12-06T09:27:12.448736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.490909+0000","last_change":"2023-12-06T09:23:45.621641+0000","last_active":"2023-12-06T09:27:12.490909+0000","last_peered":"2023-12-06T09:27:12.490909+0000","last_clean":"2023-12-06T09:27:12.490909+0000","last_became_active":"2023-12-06T09:23:45.615923+0000","last_became_peered":"2023-12-06T09:23:45.615923+0000","last_unstale":"2023-12-06T09:27:12.490909+0000","last_undegraded":"2023-12-06T09:27:12.490909+0000","last_fullsized":"2023-12-06T09:27:12.490909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:06:07.386710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":87,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-06T09:27:15.686757+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:27:15.686757+0000","last_peered":"2023-12-06T09:27:15.686757+0000","last_clean":"2023-12-06T09:27:15.686757+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:27:15.686757+0000","last_undegraded":"2023-12-06T09:27:15.686757+0000","last_fullsized":"2023-12-06T09:27:15.686757+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.493466+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:27:12.493466+0000","last_peered":"2023-12-06T09:27:12.493466+0000","last_clean":"2023-12-06T09:27:12.493466+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:27:12.493466+0000","last_undegraded":"2023-12-06T09:27:12.493466+0000","last_fullsized":"2023-12-06T09:27:12.493466+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":201,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:13.972479+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:27:13.972479+0000","last_peered":"2023-12-06T09:27:13.972479+0000","last_clean":"2023-12-06T09:27:13.972479+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:27:13.972479+0000","last_undegraded":"2023-12-06T09:27:13.972479+0000","last_fullsized":"2023-12-06T09:27:13.972479+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509507,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":737280,"data_stored":274958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":9.5999999999999996,"5min":10.960000000000001,"15min":14.996},"min":{"1min":4.4269999999999996,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":20.725999999999999,"5min":49.691000000000003,"15min":49.691000000000003},"last":17.021000000000001},{"interface":"front","average":{"1min":8.8949999999999996,"5min":10.445,"15min":14.975},"min":{"1min":5.8170000000000002,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":16.494,"5min":51.530000000000001,"15min":51.530000000000001},"last":7.6900000000000004}]},{"osd":1,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.195,"5min":10.938000000000001,"15min":12.022},"min":{"1min":5.7530000000000001,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":40.991,"5min":40.991,"15min":40.991},"last":8.4499999999999993},{"interface":"front","average":{"1min":15.298,"5min":13.292,"15min":14.442},"min":{"1min":8.1880000000000006,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":35.850000000000001,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.501999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.053000000000001,"5min":12.885,"15min":14.698},"min":{"1min":7.3819999999999997,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":18.181999999999999,"5min":42.523000000000003,"15min":42.523000000000003},"last":14.212},{"interface":"front","average":{"1min":14.087,"5min":15.260999999999999,"15min":17.285},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":21.725999999999999,"5min":39.066000000000003,"15min":39.066000000000003},"last":6.7569999999999997}]},{"osd":4,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.579000000000001,"5min":13.173,"15min":14.601000000000001},"min":{"1min":9.1319999999999997,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":34.061,"5min":34.061,"15min":34.061},"last":22.126999999999999},{"interface":"front","average":{"1min":19.207999999999998,"5min":15.503,"15min":15.571},"min":{"1min":11.127000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":32.462000000000003,"5min":32.462000000000003,"15min":32.462000000000003},"last":21.379999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.001999999999999,"5min":16.791,"15min":24.204999999999998},"min":{"1min":8.8490000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":24.170999999999999,"5min":190.821,"15min":190.821},"last":20.484999999999999},{"interface":"front","average":{"1min":17.789999999999999,"5min":18.347000000000001,"15min":24.542999999999999},"min":{"1min":6.1950000000000003,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":44.625999999999998,"5min":191.86199999999999,"15min":191.86199999999999},"last":25.943999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":20.004000000000001,"5min":18.93,"15min":24.388999999999999},"min":{"1min":11.297000000000001,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":33.362000000000002,"5min":166.45400000000001,"15min":166.45400000000001},"last":18.919},{"interface":"front","average":{"1min":18.824999999999999,"5min":19.602,"15min":24.550000000000001},"min":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.606000000000002,"5min":117.58499999999999,"15min":117.58499999999999},"last":25.146999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":23.681000000000001,"5min":23.681000000000001,"15min":23.681000000000001},"min":{"1min":12.381,"5min":12.381,"15min":12.381},"max":{"1min":52.109000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":22.834},{"interface":"front","average":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"min":{"1min":11.01,"5min":11.01,"15min":11.01},"max":{"1min":56.914000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":28.093}]}]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.0969999999999995,"15min":8.1890000000000001},"min":{"1min":6.0819999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":25.431000000000001,"5min":38.481000000000002,"15min":38.481000000000002},"last":8.1489999999999991},{"interface":"front","average":{"1min":10.859,"5min":11.725,"15min":12.146000000000001},"min":{"1min":5.7930000000000001,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":26.995000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":6.9189999999999996}]},{"osd":1,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":13.384,"15min":14.712},"min":{"1min":9.0299999999999994,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":26.780000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":10.257},{"interface":"front","average":{"1min":16.044,"5min":16.731999999999999,"15min":18.222000000000001},"min":{"1min":9.6699999999999999,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":28.262,"5min":54.338000000000001,"15min":54.338000000000001},"last":11.132}]},{"osd":3,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":11.717000000000001,"15min":11.206},"min":{"1min":7.7359999999999998,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":21.158000000000001,"5min":27.295999999999999,"15min":27.295999999999999},"last":32.860999999999997},{"interface":"front","average":{"1min":11.641,"5min":11.231,"15min":12.781000000000001},"min":{"1min":6.468,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":19.015999999999998,"5min":28.669,"15min":28.669},"last":11.763999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.824,"5min":14.007,"15min":14.811},"min":{"1min":7.2889999999999997,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":20.074999999999999,"5min":54.32,"15min":54.32},"last":19.866},{"interface":"front","average":{"1min":15.898,"5min":16.734000000000002,"15min":17.497},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":26.274000000000001,"5min":51.950000000000003,"15min":51.950000000000003},"last":12.297000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.038,"15min":15.157},"min":{"1min":7.0330000000000004,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":48.563000000000002,"5min":56.491,"15min":56.491},"last":21.747},{"interface":"front","average":{"1min":16.576000000000001,"5min":16.488,"15min":19.004000000000001},"min":{"1min":9.5960000000000001,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":47.328000000000003,"5min":75.540999999999997,"15min":75.540999999999997},"last":22.581}]},{"osd":6,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.277999999999999,"15min":18.053000000000001},"min":{"1min":10.872999999999999,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":28.962,"5min":30.213999999999999,"15min":30.213999999999999},"last":13.906000000000001},{"interface":"front","average":{"1min":14.414,"5min":13.6,"15min":14.318},"min":{"1min":8.2479999999999993,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":24.893999999999998,"5min":26.312999999999999,"15min":26.312999999999999},"last":24.859999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:58 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":17.641999999999999,"15min":17.641999999999999},"min":{"1min":12.304,"5min":12.304,"15min":12.304},"max":{"1min":30.283000000000001,"5min":30.283000000000001,"15min":30.283000000000001},"last":20.814},{"interface":"front","average":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":19.783000000000001},"min":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":31.402999999999999,"5min":31.402999999999999,"15min":31.402999999999999},"last":23.962}]}]},{"osd":0,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":270541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.576000000000001,"5min":11.565,"15min":11.337999999999999},"min":{"1min":4.8289999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":52.207000000000001,"5min":65.808000000000007,"15min":65.808000000000007},"last":8.4589999999999996},{"interface":"front","average":{"1min":12.784000000000001,"5min":12.163,"15min":11.606999999999999},"min":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":53.106999999999999,"5min":53.106999999999999,"15min":53.106999999999999},"last":13.702}]},{"osd":2,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.065,"15min":13.079000000000001},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":41.301000000000002,"5min":49.652999999999999,"15min":49.652999999999999},"last":17.263000000000002},{"interface":"front","average":{"1min":12.539,"5min":13.265000000000001,"15min":14.048},"min":{"1min":7.9960000000000004,"5min":6.798,"15min":6.798},"max":{"1min":21.843,"5min":47.484000000000002,"15min":47.484000000000002},"last":9.2100000000000009}]},{"osd":3,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.029,"5min":13.706,"15min":20.071999999999999},"min":{"1min":6.7320000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":32.063000000000002,"5min":205.096,"15min":205.096},"last":11.638999999999999},{"interface":"front","average":{"1min":18.690000000000001,"5min":17.489000000000001,"15min":22.172000000000001},"min":{"1min":8.5660000000000007,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":45.463000000000001,"5min":182.74000000000001,"15min":182.74000000000001},"last":16.628}]},{"osd":4,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.943,"5min":13.228,"15min":13.978},"min":{"1min":8.0660000000000007,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":29.585999999999999,"5min":45.112000000000002,"15min":45.112000000000002},"last":31.451000000000001},{"interface":"front","average":{"1min":17.719000000000001,"5min":16.870999999999999,"15min":16.803000000000001},"min":{"1min":7.1470000000000002,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":34.533000000000001,"5min":52.655999999999999,"15min":52.655999999999999},"last":30.16}]},{"osd":5,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":15.237,"5min":15.542,"15min":15.226000000000001},"min":{"1min":9.2129999999999992,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":25.379999999999999,"5min":43.088000000000001,"15min":43.088000000000001},"last":10.427},{"interface":"front","average":{"1min":16.452999999999999,"5min":16.266999999999999,"15min":16.77},"min":{"1min":9.7230000000000008,"5min":8.266,"15min":8.266},"max":{"1min":31.550000000000001,"5min":45.722000000000001,"15min":45.722000000000001},"last":7.7709999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":21.395,"5min":19.311,"15min":20.494},"min":{"1min":9.2309999999999999,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":44.292999999999999,"5min":78.338999999999999,"15min":78.338999999999999},"last":27.809000000000001},{"interface":"front","average":{"1min":18.216000000000001,"5min":17.864999999999998,"15min":20.274000000000001},"min":{"1min":10.634,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":42.417999999999999,"5min":80.850999999999999,"15min":80.850999999999999},"last":15.49}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":18.908999999999999,"5min":18.908999999999999,"15min":18.908999999999999},"min":{"1min":8.2469999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":61.445999999999998,"5min":61.445999999999998,"15min":61.445999999999998},"last":12.217000000000001},{"interface":"front","average":{"1min":20.067,"5min":20.067,"15min":20.067},"min":{"1min":11.833,"5min":11.833,"15min":11.833},"max":{"1min":63.518000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":14.566000000000001}]}]},{"osd":7,"up_from":60,"seq":257698037778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28692,"kb_used_data":708,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343148,"statfs":{"total":96636764160,"available":96607383552,"internally_reserved":0,"allocated":724992,"data_stored":270541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":11.984999999999999,"5min":11.984999999999999,"15min":11.984999999999999},"min":{"1min":4.452,"5min":4.452,"15min":4.452},"max":{"1min":55.622,"5min":55.622,"15min":55.622},"last":6.984},{"interface":"front","average":{"1min":12.089,"5min":12.089,"15min":12.089},"min":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":31.178000000000001,"5min":31.178000000000001,"15min":31.178000000000001},"last":25.652999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":48.959000000000003,"5min":48.959000000000003,"15min":48.959000000000003},"last":28.062999999999999},{"interface":"front","average":{"1min":13.994999999999999,"5min":13.994999999999999,"15min":13.994999999999999},"min":{"1min":8.0380000000000003,"5min":8.0380000000000003,"15min":8.0380000000000003},"max":{"1min":38.232999999999997,"5min":38.232999999999997,"15min":38.232999999999997},"last":13.35}]},{"osd":2,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":12.195,"5min":12.195,"15min":12.195},"min":{"1min":5.952,"5min":5.952,"15min":5.952},"max":{"1min":20.297999999999998,"5min":20.297999999999998,"15min":20.297999999999998},"last":22.370000000000001},{"interface":"front","average":{"1min":15.340999999999999,"5min":15.340999999999999,"15min":15.340999999999999},"min":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":48.240000000000002,"5min":48.240000000000002,"15min":48.240000000000002},"last":29.916}]},{"osd":3,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":13.568,"15min":13.568},"min":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"max":{"1min":54.603999999999999,"5min":54.603999999999999,"15min":54.603999999999999},"last":20.783000000000001},{"interface":"front","average":{"1min":21.209,"5min":21.209,"15min":21.209},"min":{"1min":12.032999999999999,"5min":12.032999999999999,"15min":12.032999999999999},"max":{"1min":52.719000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":23.792000000000002}]},{"osd":4,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.044,"5min":15.044,"15min":15.044},"min":{"1min":7.2329999999999997,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":32.390999999999998,"5min":32.390999999999998,"15min":32.390999999999998},"last":30.542000000000002},{"interface":"front","average":{"1min":13.667999999999999,"5min":13.667999999999999,"15min":13.667999999999999},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":23.030999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.632999999999999,"5min":19.632999999999999,"15min":19.632999999999999},"min":{"1min":12.234,"5min":12.234,"15min":12.234},"max":{"1min":52.197000000000003,"5min":52.197000000000003,"15min":52.197000000000003},"last":26.657},{"interface":"front","average":{"1min":17.359999999999999,"5min":17.359999999999999,"15min":17.359999999999999},"min":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":38.877000000000002,"5min":38.877000000000002,"15min":38.877000000000002},"last":31.157}]},{"osd":6,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.231000000000002,"5min":19.231000000000002,"15min":19.231000000000002},"min":{"1min":12.494999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":39.508000000000003,"5min":39.508000000000003,"15min":39.508000000000003},"last":32.341000000000001},{"interface":"front","average":{"1min":14.912000000000001,"5min":14.912000000000001,"15min":14.912000000000001},"min":{"1min":8.6530000000000005,"5min":8.6530000000000005,"15min":8.6530000000000005},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":21.428000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":8.9559999999999995,"5min":10.226000000000001,"15min":11.521000000000001},"min":{"1min":4.3490000000000002,"5min":3.484,"15min":3.484},"max":{"1min":16.178999999999998,"5min":45.506,"15min":45.506},"last":11.682},{"interface":"front","average":{"1min":9.0839999999999996,"5min":10.619999999999999,"15min":12.609999999999999},"min":{"1min":5.1120000000000001,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":16.661000000000001,"5min":46.450000000000003,"15min":46.450000000000003},"last":8.1010000000000009}]},{"osd":1,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":11.772,"5min":13.038,"15min":15.023999999999999},"min":{"1min":7.4740000000000002,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":20.497,"5min":55.398000000000003,"15min":55.398000000000003},"last":10.119999999999999},{"interface":"front","average":{"1min":13.172000000000001,"5min":13.750999999999999,"15min":14.923},"min":{"1min":8.9809999999999999,"5min":6.569,"15min":6.569},"max":{"1min":22.056999999999999,"5min":41.811999999999998,"15min":41.811999999999998},"last":17.503}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":15.488,"15min":20.422000000000001},"min":{"1min":11.131,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":23.66,"5min":208.93799999999999,"15min":208.93799999999999},"last":18.369},{"interface":"front","average":{"1min":13.587,"5min":16.247,"15min":24.045999999999999},"min":{"1min":7.3310000000000004,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":24.765000000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":13.029999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":17.373999999999999,"5min":15.544,"15min":17.196999999999999},"min":{"1min":10.898999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":25.806999999999999,"5min":43.276000000000003,"15min":43.276000000000003},"last":24.34},{"interface":"front","average":{"1min":9.9079999999999995,"5min":10.916,"15min":14.124000000000001},"min":{"1min":6.4320000000000004,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":17.137,"5min":41.493000000000002,"15min":41.493000000000002},"last":10.882999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":16.132000000000001,"5min":16.742999999999999,"15min":22.754999999999999},"min":{"1min":9.4000000000000004,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":27.443999999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":15.029999999999999},{"interface":"front","average":{"1min":18.768000000000001,"5min":18.789000000000001,"15min":25.478000000000002},"min":{"1min":8.4570000000000007,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":29.385000000000002,"5min":111.874,"15min":111.874},"last":25.707000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":12.522,"5min":12.273,"15min":16.116},"min":{"1min":7.3929999999999998,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":22.010000000000002,"5min":39.886000000000003,"15min":39.886000000000003},"last":12.359999999999999},{"interface":"front","average":{"1min":17.369,"5min":17.016999999999999,"15min":21.588999999999999},"min":{"1min":11.013999999999999,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":30.199999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":16.481000000000002}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":20.678999999999998,"5min":20.678999999999998,"15min":20.678999999999998},"min":{"1min":13.064,"5min":13.064,"15min":13.064},"max":{"1min":31.757999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":27.405000000000001},{"interface":"front","average":{"1min":15.526999999999999,"5min":15.526999999999999,"15min":15.526999999999999},"min":{"1min":9.6890000000000001,"5min":9.6890000000000001,"15min":9.6890000000000001},"max":{"1min":28.809000000000001,"5min":28.809000000000001,"15min":28.809000000000001},"last":29.145}]}]},{"osd":5,"up_from":14,"seq":60129542210,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":663964,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":9.0600000000000005,"5min":10.461,"15min":13.067},"min":{"1min":4.5670000000000002,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":18.443000000000001,"5min":45.491,"15min":45.491},"last":38.012},{"interface":"front","average":{"1min":8.7940000000000005,"5min":10.765000000000001,"15min":12.897},"min":{"1min":4.4349999999999996,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":14.475,"5min":49.991999999999997,"15min":49.991999999999997},"last":36.920999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":11.146000000000001,"5min":13.614000000000001,"15min":21.236000000000001},"min":{"1min":5.6900000000000004,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":18.754000000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":16.754000000000001},{"interface":"front","average":{"1min":10.789,"5min":17.363,"15min":29.614999999999998},"min":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":17.491,"5min":344.79599999999999,"15min":344.79599999999999},"last":10.994999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.129,"5min":12.884,"15min":13.379},"min":{"1min":9.1910000000000007,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":24.969999999999999,"5min":53.235999999999997,"15min":53.235999999999997},"last":7.3490000000000002},{"interface":"front","average":{"1min":14.458,"5min":13.545999999999999,"15min":15.108000000000001},"min":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":26.155000000000001,"5min":54.747,"15min":54.747},"last":15.968}]},{"osd":3,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.762,"5min":14.917,"15min":13.975},"min":{"1min":9.6069999999999993,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":24.18,"5min":55.310000000000002,"15min":55.310000000000002},"last":15.468999999999999},{"interface":"front","average":{"1min":11.239000000000001,"5min":12.801,"15min":16.099},"min":{"1min":6.9889999999999999,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":19.765000000000001,"5min":51.183,"15min":51.183},"last":13.975}]},{"osd":4,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":12.808,"5min":14.959,"15min":19.792999999999999},"min":{"1min":9.0570000000000004,"5min":6.085,"15min":6.085},"max":{"1min":21.306999999999999,"5min":281.608,"15min":281.608},"last":8.375},{"interface":"front","average":{"1min":16.542999999999999,"5min":16.850999999999999,"15min":21.760999999999999},"min":{"1min":11.683999999999999,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":23.295999999999999,"5min":225.83600000000001,"15min":225.83600000000001},"last":11.868}]},{"osd":6,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":16.638999999999999,"5min":16.809000000000001,"15min":20.645},"min":{"1min":11.105,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":22.853000000000002,"5min":198.667,"15min":198.667},"last":12.465999999999999},{"interface":"front","average":{"1min":16.501000000000001,"5min":17.471,"15min":22.067},"min":{"1min":10.128,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":24.225000000000001,"5min":208.446,"15min":208.446},"last":17.536000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":14.349,"5min":14.349,"15min":14.349},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":9.9860000000000007},{"interface":"front","average":{"1min":18.501000000000001,"5min":18.501000000000001,"15min":18.501000000000001},"min":{"1min":12.881,"5min":12.881,"15min":12.881},"max":{"1min":25.683,"5min":25.683,"15min":25.683},"last":20.109000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542210,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":737280,"data_stored":659547,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.8759999999999994,"5min":8.9649999999999999,"15min":10.57},"min":{"1min":5.6459999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":26.015000000000001,"5min":40.225999999999999,"15min":40.225999999999999},"last":16.841999999999999},{"interface":"front","average":{"1min":11.209,"5min":10.295999999999999,"15min":11.488},"min":{"1min":5.8639999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":20.613,"5min":30.405000000000001,"15min":30.405000000000001},"last":5.3319999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":10.425000000000001,"5min":13.679,"15min":11.686},"min":{"1min":7.0549999999999997,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":21.376000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":17.466000000000001},{"interface":"front","average":{"1min":15.489000000000001,"5min":17.225999999999999,"15min":16.045999999999999},"min":{"1min":10.004,"5min":6.415,"15min":6.415},"max":{"1min":30.391999999999999,"5min":164.911,"15min":164.911},"last":22.452999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.678000000000001,"5min":14.433999999999999,"15min":19.654},"min":{"1min":5.359,"5min":5.359,"15min":5.359},"max":{"1min":20.265999999999998,"5min":122.762,"15min":122.762},"last":8.1989999999999998},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.653,"15min":19.021000000000001},"min":{"1min":7.1059999999999999,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":22.109000000000002,"5min":129.99100000000001,"15min":129.99100000000001},"last":23.276}]},{"osd":4,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.468999999999999,"5min":18.315000000000001,"15min":27.951000000000001},"min":{"1min":9.1140000000000008,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":49.097000000000001,"5min":359.46300000000002,"15min":359.46300000000002},"last":9.1699999999999999},{"interface":"front","average":{"1min":15.288,"5min":17.658999999999999,"15min":26.722999999999999},"min":{"1min":6.4539999999999997,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":47.192,"5min":314.47699999999998,"15min":314.47699999999998},"last":19.954999999999998}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":19.167999999999999,"5min":21.059999999999999,"15min":25.846},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":81.923000000000002,"5min":152.97399999999999,"15min":152.97399999999999},"last":20.901},{"interface":"front","average":{"1min":16.448,"5min":18.698,"15min":24.085000000000001},"min":{"1min":10.108000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.151000000000003,"5min":154.53100000000001,"15min":154.53100000000001},"last":18.18}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.266999999999999,"5min":15.220000000000001,"15min":17.032},"min":{"1min":8.218,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":20.634,"5min":43.045000000000002,"15min":43.045000000000002},"last":24.062000000000001},{"interface":"front","average":{"1min":16.388000000000002,"5min":17.977,"15min":20.286999999999999},"min":{"1min":11.009,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":27.966999999999999,"5min":41.356000000000002,"15min":41.356000000000002},"last":15.733000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":19.687999999999999,"15min":19.687999999999999},"min":{"1min":11.547000000000001,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":32.191000000000003,"5min":32.191000000000003,"15min":32.191000000000003},"last":24.834},{"interface":"front","average":{"1min":19.468,"5min":19.468,"15min":19.468},"min":{"1min":13.093,"5min":13.093,"15min":13.093},"max":{"1min":36.454000000000001,"5min":36.454000000000001,"15min":36.454000000000001},"last":26.558}]}]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5269999999999992,"5min":11.606,"15min":13.595000000000001},"min":{"1min":4.9740000000000002,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":23.562000000000001,"5min":137.13999999999999,"15min":137.13999999999999},"last":7.1269999999999998},{"interface":"front","average":{"1min":9.0630000000000006,"5min":12.724,"15min":17.661999999999999},"min":{"1min":4.1959999999999997,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":23.231000000000002,"5min":199.268,"15min":199.268},"last":13.186999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.608000000000001,"15min":18.324000000000002},"min":{"1min":5.2779999999999996,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":25.949999999999999,"5min":92.817999999999998,"15min":92.817999999999998},"last":10.518000000000001},{"interface":"front","average":{"1min":11.914,"5min":14.56,"15min":19.475999999999999},"min":{"1min":7.8040000000000003,"5min":5.556,"15min":5.556},"max":{"1min":17.295000000000002,"5min":95.665999999999997,"15min":95.665999999999997},"last":5.6580000000000004}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":13.292999999999999,"5min":17.664000000000001,"15min":23.103999999999999},"min":{"1min":7.4470000000000001,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":23.303999999999998,"5min":208.06,"15min":208.06},"last":14.835000000000001},{"interface":"front","average":{"1min":13.791,"5min":16.366,"15min":18.635999999999999},"min":{"1min":8.7569999999999997,"5min":6.915,"15min":6.915},"max":{"1min":21.555,"5min":147.18199999999999,"15min":147.18199999999999},"last":24.934999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":13.999000000000001,"15min":18.155000000000001},"min":{"1min":6.2610000000000001,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":28.175999999999998,"5min":121.627,"15min":121.627},"last":12.118},{"interface":"front","average":{"1min":15.657999999999999,"5min":16.553999999999998,"15min":18.242000000000001},"min":{"1min":10.692,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":25.760999999999999,"5min":74.602000000000004,"15min":74.602000000000004},"last":11.311}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":16.504000000000001,"5min":16.984000000000002,"15min":18.885000000000002},"min":{"1min":8.3930000000000007,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":24.431000000000001,"5min":71.759,"15min":71.759},"last":16.286999999999999},{"interface":"front","average":{"1min":13.960000000000001,"5min":14.284000000000001,"15min":16.748000000000001},"min":{"1min":6.6399999999999997,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":29.401,"5min":70.882999999999996,"15min":70.882999999999996},"last":14.121}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.581,"5min":16.417999999999999,"15min":19.870000000000001},"min":{"1min":8.9700000000000006,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":22.050999999999998,"5min":80.668999999999997,"15min":80.668999999999997},"last":25.928000000000001},{"interface":"front","average":{"1min":17.324999999999999,"5min":19.062999999999999,"15min":21.738},"min":{"1min":12.243,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":25.762,"5min":72.796999999999997,"15min":72.796999999999997},"last":18.181000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":17.497,"5min":17.497,"15min":17.497},"min":{"1min":11.442,"5min":11.442,"15min":11.442},"max":{"1min":29.594000000000001,"5min":29.594000000000001,"15min":29.594000000000001},"last":26.902999999999999},{"interface":"front","average":{"1min":18.670999999999999,"5min":18.670999999999999,"15min":18.670999999999999},"min":{"1min":11.663,"5min":11.663,"15min":11.663},"max":{"1min":27.414000000000001,"5min":27.414000000000001,"15min":27.414000000000001},"last":17.550999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:27:19.403 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-06T09:27:19.404 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:27:19.404 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T09:27:19.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T09:27:19.466 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:19.467 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:19.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:19.486 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:19.500+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:19.585 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:19.599+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:19.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:19.648 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:19.648 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:19.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:19.698 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:19.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:19.787 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:19.801+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:19.831 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:19.831 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:19.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:19.925 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:19.939+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:20.011 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:20.011 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:20.020 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:20.021 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:20.021 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:20.035+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:20.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:20.090 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:20.104+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:20.107 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:20.107 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:20.188 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:20.189 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:20.191 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:20.205+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:20.196 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:20.196 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:20.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:20.226 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:27:20.226 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "epoch": 85, 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "fsid": "0c48cca3-dcc0-44bd-8eed-37fea2138550", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "created": "2023-12-06T09:21:08.071156+0000", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "modified": "2023-12-06T09:27:17.930307+0000", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "last_up_change": "2023-12-06T09:25:55.041559+0000", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "last_in_change": "2023-12-06T09:24:35.591387+0000", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_num": 5799936, 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_set": [ 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "pglog_hardlimit", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "purged_snapdirs", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_deletes", 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: "sortbitwise" 2023-12-06T09:27:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_version": 7, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_snapshot": "", 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_max": 2, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "max_osd": 8, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "require_min_compat_client": "luminous", 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "min_compat_client": "luminous", 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "require_osd_release": "reef", 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "allow_crimson": false, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "pools": [ 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 1, 2023-12-06T09:27:20.228 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": ".mgr", 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:21:54.386974+0000", 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 1, 2023-12-06T09:27:20.229 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 1, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 1, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 1, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 1, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "18", 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:27:20.230 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:27:20.231 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:27:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "options": { 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_max": 32, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_min": 1 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": {} 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 8, 2023-12-06T09:27:20.233 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 8, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.25, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 2, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 2, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 0.82869338989257812, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 0.82999998331069946, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "pool": 2, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_name": "rbd", 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "create_time": "2023-12-06T09:22:01.832571+0000", 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "flags": 1, 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "flags_names": "hashpspool", 2023-12-06T09:27:20.234 INFO:teuthology.orchestra.run.smithi088.stdout: "type": 1, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 2, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "min_size": 1, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_rule": 0, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_count": 0, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_target": 0, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "object_hash": 2, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_autoscale_mode": "off", 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num": 8, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num": 8, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_placement_num_target": 8, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_target": 8, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_num_pending": 8, 2023-12-06T09:27:20.235 INFO:teuthology.orchestra.run.smithi088.stdout: "last_pg_merge_meta": { 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "source_pgid": "0.0", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "ready_epoch": 0, 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_started": 0, 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "last_epoch_clean": 0, 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "source_version": "0'0", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "target_version": "0'0" 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "last_change": "21", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend": "0", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "auid": 0, 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_mode": "selfmanaged", 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_seq": 0, 2023-12-06T09:27:20.236 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_epoch": 0, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "pool_snaps": [], 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps": "[]", 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_bytes": 0, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "quota_max_objects": 0, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "tiers": [], 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "tier_of": -1, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "read_tier": -1, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "write_tier": -1, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_mode": "none", 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_bytes": 0, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "target_max_objects": 0, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T09:27:20.237 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_flush_age": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "cache_min_evict_age": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profile": "", 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_params": { 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "none" 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_period": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_count": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "use_gmt_hitset": true, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "min_read_recency_for_promote": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "min_write_recency_for_promote": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "hit_set_search_last_n": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "grade_table": [], 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "stripe_width": 0, 2023-12-06T09:27:20.238 INFO:teuthology.orchestra.run.smithi088.stdout: "expected_num_objects": 0, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "fast_read": false, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "options": {}, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "application_metadata": { 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "rbd": {} 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "read_balance": { 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "score_acting": 1.9900000095367432, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "score_stable": 1.9900000095367432, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_acting": 1.75, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "raw_score_stable": 1.75, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity_weighted": 0.82869338989257812, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity": 0.82999998331069946, 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2023-12-06T09:27:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "db81b709-a318-4b43-beb2-bf3687f365e1", 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 0.4708404541015625, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 52, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.240 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6805", 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6806", 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6808", 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6807", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2388339328 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6805/2388339328", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6806/2388339328", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6808/2388339328", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6807/2388339328", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.243 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd7b0482-e237-424b-8eff-8db905713049", 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:27:20.244 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 49, 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6801", 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.245 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6802", 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6804", 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:27:20.246 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6803", 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3646708602 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6801/3646708602", 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6802/3646708602", 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6804/3646708602", 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6803/3646708602", 2023-12-06T09:27:20.247 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e5a571be-6b8e-4c6f-bd3e-2af30c112285", 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:27:20.248 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 41, 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6810", 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.249 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6813", 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6816", 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.250 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6814", 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3714168299 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6810/3714168299", 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6813/3714168299", 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6816/3714168299", 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6814/3714168299", 2023-12-06T09:27:20.251 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "d7abe19a-cc3d-411f-8533-8592e1379735", 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 0.1587066650390625, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 15, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 49, 2023-12-06T09:27:20.252 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6809", 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.253 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6811", 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6815", 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.254 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.88:6812", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1828251745 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.88:6809/1828251745", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.88:6811/1828251745", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1828251745", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.88:6812/1828251745", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.255 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b9517160-d081-44f9-aea4-6d2529e02114", 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 13, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 44, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.256 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6800", 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6801", 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.257 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6803", 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.258 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6802", 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3720978101 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6800/3720978101", 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6801/3720978101", 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6803/3720978101", 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6802/3720978101", 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.259 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e", 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 49, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6804", 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6805", 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.261 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6807", 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6806", 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1203998358 2023-12-06T09:27:20.262 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6804/1203998358", 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6805/1203998358", 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6807/1203998358", 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6806/1203998358", 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "e7afeca8-1363-4e9b-96f6-7fab6113d9e0", 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.263 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 0, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 0, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 14, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 46, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 0, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6808", 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:27:20.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6809", 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6811", 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.266 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6810", 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 4151455937 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6808/4151455937", 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6809/4151455937", 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6811/4151455937", 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6810/4151455937", 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "uuid": "fd4b3971-3adb-44fd-b42b-b6660305beb3", 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "up": 1, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "in": 1, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "weight": 1, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_affinity": 1, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_begin": 14, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "last_clean_end": 51, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "up_from": 60, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "up_thru": 60, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "down_at": 52, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "lost_at": 0, 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addrs": { 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.268 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6812", 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1573021680 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addrs": { 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6813", 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1573021680 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addrs": { 2023-12-06T09:27:20.269 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6815", 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1573021680 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addrs": { 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "addrvec": [ 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v2", 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "172.21.15.132:6814", 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1573021680 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.270 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "public_addr": "172.21.15.132:6812/1573021680", 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_addr": "172.21.15.132:6813/1573021680", 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_back_addr": "172.21.15.132:6815/1573021680", 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "heartbeat_front_addr": "172.21.15.132:6814/1573021680", 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "state": [ 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "exists", 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: "up" 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.271 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_xinfo": [ 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 0, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.229602+0000", 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 1, 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.272 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.158659+0000", 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 2, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.969728+0000", 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 3, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:53.234010+0000", 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 4, 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.274 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:50.601730+0000", 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 5, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.304649+0000", 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 6, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:51.671399+0000", 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 0 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": 7, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "down_stamp": "0.000000", 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_probability": 0, 2023-12-06T09:27:20.276 INFO:teuthology.orchestra.run.smithi088.stdout: "laggy_interval": 0, 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "features": 4540138322906710015, 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "old_weight": 0, 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "last_purged_snaps_scrub": "2023-12-06T09:21:52.055036+0000", 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "dead_epoch": 51 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap": [ 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "pgid": "2.1", 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: 3, 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: 7 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: "pgid": "2.7", 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: "osds": [ 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: 7, 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: 0 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_items": [ 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: "pgid": "2.1", 2023-12-06T09:27:20.278 INFO:teuthology.orchestra.run.smithi088.stdout: "mappings": [ 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "from": 7, 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "to": 2 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_upmap_primaries": [], 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pg_temp": [], 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "primary_temp": [], 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklist": {}, 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "range_blocklist": {}, 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "erasure_code_profiles": { 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "default": { 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "crush-failure-domain": "osd", 2023-12-06T09:27:20.279 INFO:teuthology.orchestra.run.smithi088.stdout: "k": "2", 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "m": "1", 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "plugin": "jerasure", 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "technique": "reed_sol_van" 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "removed_snaps_queue": [], 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "new_removed_snaps": [], 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "new_purged_snaps": [], 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "crush_node_flags": {}, 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "device_class_flags": {}, 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode": { 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_enabled": false, 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_bucket_count": 0, 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "degraded_stretch_mode": 0, 2023-12-06T09:27:20.280 INFO:teuthology.orchestra.run.smithi088.stdout: "recovering_stretch_mode": 0, 2023-12-06T09:27:20.281 INFO:teuthology.orchestra.run.smithi088.stdout: "stretch_mode_bucket": 0 2023-12-06T09:27:20.281 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:20.281 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:20.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:27:20.283 INFO:tasks.thrashosds.thrasher:j is {'epoch': 85, 'fsid': '0c48cca3-dcc0-44bd-8eed-37fea2138550', 'created': '2023-12-06T09:21:08.071156+0000', 'modified': '2023-12-06T09:27:17.930307+0000', 'last_up_change': '2023-12-06T09:25:55.041559+0000', 'last_in_change': '2023-12-06T09:24:35.591387+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T09:21:54.386974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8286933898925781, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T09:22:01.832571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8286933898925781, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'db81b709-a318-4b43-beb2-bf3687f365e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4708404541015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6805', 'nonce': 2388339328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6806', 'nonce': 2388339328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6808', 'nonce': 2388339328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6807', 'nonce': 2388339328}]}, 'public_addr': '172.21.15.88:6805/2388339328', 'cluster_addr': '172.21.15.88:6806/2388339328', 'heartbeat_back_addr': '172.21.15.88:6808/2388339328', 'heartbeat_front_addr': '172.21.15.88:6807/2388339328', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'fd7b0482-e237-424b-8eff-8db905713049', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 49, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6801', 'nonce': 3646708602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6802', 'nonce': 3646708602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6804', 'nonce': 3646708602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6803', 'nonce': 3646708602}]}, 'public_addr': '172.21.15.88:6801/3646708602', 'cluster_addr': '172.21.15.88:6802/3646708602', 'heartbeat_back_addr': '172.21.15.88:6804/3646708602', 'heartbeat_front_addr': '172.21.15.88:6803/3646708602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e5a571be-6b8e-4c6f-bd3e-2af30c112285', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6810', 'nonce': 3714168299}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6813', 'nonce': 3714168299}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6816', 'nonce': 3714168299}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6814', 'nonce': 3714168299}]}, 'public_addr': '172.21.15.88:6810/3714168299', 'cluster_addr': '172.21.15.88:6813/3714168299', 'heartbeat_back_addr': '172.21.15.88:6816/3714168299', 'heartbeat_front_addr': '172.21.15.88:6814/3714168299', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd7abe19a-cc3d-411f-8533-8592e1379735', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1587066650390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 49, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6809', 'nonce': 1828251745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6811', 'nonce': 1828251745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6815', 'nonce': 1828251745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6812', 'nonce': 1828251745}]}, 'public_addr': '172.21.15.88:6809/1828251745', 'cluster_addr': '172.21.15.88:6811/1828251745', 'heartbeat_back_addr': '172.21.15.88:6815/1828251745', 'heartbeat_front_addr': '172.21.15.88:6812/1828251745', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b9517160-d081-44f9-aea4-6d2529e02114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 44, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6800', 'nonce': 3720978101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6801', 'nonce': 3720978101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6803', 'nonce': 3720978101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6802', 'nonce': 3720978101}]}, 'public_addr': '172.21.15.132:6800/3720978101', 'cluster_addr': '172.21.15.132:6801/3720978101', 'heartbeat_back_addr': '172.21.15.132:6803/3720978101', 'heartbeat_front_addr': '172.21.15.132:6802/3720978101', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 49, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6804', 'nonce': 1203998358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6805', 'nonce': 1203998358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6807', 'nonce': 1203998358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6806', 'nonce': 1203998358}]}, 'public_addr': '172.21.15.132:6804/1203998358', 'cluster_addr': '172.21.15.132:6805/1203998358', 'heartbeat_back_addr': '172.21.15.132:6807/1203998358', 'heartbeat_front_addr': '172.21.15.132:6806/1203998358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7afeca8-1363-4e9b-96f6-7fab6113d9e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6808', 'nonce': 4151455937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6809', 'nonce': 4151455937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6811', 'nonce': 4151455937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6810', 'nonce': 4151455937}]}, 'public_addr': '172.21.15.132:6808/4151455937', 'cluster_addr': '172.21.15.132:6809/4151455937', 'heartbeat_back_addr': '172.21.15.132:6811/4151455937', 'heartbeat_front_addr': '172.21.15.132:6810/4151455937', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fd4b3971-3adb-44fd-b42b-b6660305beb3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 14, 'last_clean_end': 51, 'up_from': 60, 'up_thru': 60, 'down_at': 52, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6812', 'nonce': 1573021680}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6813', 'nonce': 1573021680}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6815', 'nonce': 1573021680}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6814', 'nonce': 1573021680}]}, 'public_addr': '172.21.15.132:6812/1573021680', 'cluster_addr': '172.21.15.132:6813/1573021680', 'heartbeat_back_addr': '172.21.15.132:6815/1573021680', 'heartbeat_front_addr': '172.21.15.132:6814/1573021680', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.229602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.158659+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.969728+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:53.234010+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:50.601730+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.304649+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:51.671399+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T09:21:52.055036+0000', 'dead_epoch': 51}], 'pg_upmap': [{'pgid': '2.1', 'osds': [3, 7]}, {'pgid': '2.7', 'osds': [7, 0]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T09:27:20.283 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:27:20.292 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:20.292 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:20.293 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:20.307+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:20.337 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:20.338 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:20.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:20.391 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:20.392 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:20.393 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:20.407+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:20.487 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:20.487 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:20.494 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:20.508+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:20.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:20.506 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:20.507 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:20.594 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:20.608+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:20.599 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:20.599 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:20.649 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:20.649 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:20.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:20.701 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:20.715+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:20.707 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:20.707 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:20.787 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:20.788 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:20.788 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:20.788 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:20.788 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:20.797 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:20.811+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:20.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:20.817 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2023-12-06T09:27:20.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:27:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:27:20.903 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:20.916+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:20.916 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2023-12-06T09:27:19.974734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223036,"kb_used_data":5552,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754751684,"statfs":{"total":773094113280,"available":772865724416,"internally_reserved":0,"allocated":5685248,"data_stored":2861318,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":69,"apply_latency_ms":69,"commit_latency_ns":69000000,"apply_latency_ns":69000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.720811"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-06T09:27:16.703829+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:27:16.703829+0000","last_peered":"2023-12-06T09:27:16.703829+0000","last_clean":"2023-12-06T09:27:16.703829+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:27:16.703829+0000","last_undegraded":"2023-12-06T09:27:16.703829+0000","last_fullsized":"2023-12-06T09:27:16.703829+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-06T09:27:17.817024+0000","last_change":"2023-12-06T09:23:50.752271+0000","last_active":"2023-12-06T09:27:17.817024+0000","last_peered":"2023-12-06T09:27:17.817024+0000","last_clean":"2023-12-06T09:27:17.817024+0000","last_became_active":"2023-12-06T09:23:50.743293+0000","last_became_peered":"2023-12-06T09:23:50.743293+0000","last_unstale":"2023-12-06T09:27:17.817024+0000","last_undegraded":"2023-12-06T09:27:17.817024+0000","last_fullsized":"2023-12-06T09:27:17.817024+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:27:50.211606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-06T09:27:15.697415+0000","last_change":"2023-12-06T09:22:03.790176+0000","last_active":"2023-12-06T09:27:15.697415+0000","last_peered":"2023-12-06T09:27:15.697415+0000","last_clean":"2023-12-06T09:27:15.697415+0000","last_became_active":"2023-12-06T09:22:03.781252+0000","last_became_peered":"2023-12-06T09:22:03.781252+0000","last_unstale":"2023-12-06T09:27:15.697415+0000","last_undegraded":"2023-12-06T09:27:15.697415+0000","last_fullsized":"2023-12-06T09:27:15.697415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:39:42.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-06T09:27:16.948401+0000","last_change":"2023-12-06T09:24:37.205991+0000","last_active":"2023-12-06T09:27:16.948401+0000","last_peered":"2023-12-06T09:27:16.948401+0000","last_clean":"2023-12-06T09:27:16.948401+0000","last_became_active":"2023-12-06T09:24:37.200368+0000","last_became_peered":"2023-12-06T09:24:37.200368+0000","last_unstale":"2023-12-06T09:27:16.948401+0000","last_undegraded":"2023-12-06T09:27:16.948401+0000","last_fullsized":"2023-12-06T09:27:16.948401+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:57:43.491211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-06T09:27:17.399536+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:27:17.399536+0000","last_peered":"2023-12-06T09:27:17.399536+0000","last_clean":"2023-12-06T09:27:17.399536+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:27:17.399536+0000","last_undegraded":"2023-12-06T09:27:17.399536+0000","last_fullsized":"2023-12-06T09:27:17.399536+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.490909+0000","last_change":"2023-12-06T09:23:45.621641+0000","last_active":"2023-12-06T09:27:12.490909+0000","last_peered":"2023-12-06T09:27:12.490909+0000","last_clean":"2023-12-06T09:27:12.490909+0000","last_became_active":"2023-12-06T09:23:45.615923+0000","last_became_peered":"2023-12-06T09:23:45.615923+0000","last_unstale":"2023-12-06T09:27:12.490909+0000","last_undegraded":"2023-12-06T09:27:12.490909+0000","last_fullsized":"2023-12-06T09:27:12.490909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:06:07.386710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":87,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-06T09:27:15.686757+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:27:15.686757+0000","last_peered":"2023-12-06T09:27:15.686757+0000","last_clean":"2023-12-06T09:27:15.686757+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:27:15.686757+0000","last_undegraded":"2023-12-06T09:27:15.686757+0000","last_fullsized":"2023-12-06T09:27:15.686757+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-12-06T09:27:12.493466+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:27:12.493466+0000","last_peered":"2023-12-06T09:27:12.493466+0000","last_clean":"2023-12-06T09:27:12.493466+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:27:12.493466+0000","last_undegraded":"2023-12-06T09:27:12.493466+0000","last_fullsized":"2023-12-06T09:27:12.493466+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":203,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-06T09:27:16.716383+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:27:16.716383+0000","last_peered":"2023-12-06T09:27:16.716383+0000","last_clean":"2023-12-06T09:27:16.716383+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:27:16.716383+0000","last_undegraded":"2023-12-06T09:27:16.716383+0000","last_fullsized":"2023-12-06T09:27:16.716383+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509507,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":737280,"data_stored":274958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":9.5999999999999996,"5min":10.960000000000001,"15min":14.996},"min":{"1min":4.4269999999999996,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":20.725999999999999,"5min":49.691000000000003,"15min":49.691000000000003},"last":17.021000000000001},{"interface":"front","average":{"1min":8.8949999999999996,"5min":10.445,"15min":14.975},"min":{"1min":5.8170000000000002,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":16.494,"5min":51.530000000000001,"15min":51.530000000000001},"last":7.6900000000000004}]},{"osd":1,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.195,"5min":10.938000000000001,"15min":12.022},"min":{"1min":5.7530000000000001,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":40.991,"5min":40.991,"15min":40.991},"last":8.4499999999999993},{"interface":"front","average":{"1min":15.298,"5min":13.292,"15min":14.442},"min":{"1min":8.1880000000000006,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":35.850000000000001,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.501999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.053000000000001,"5min":12.885,"15min":14.698},"min":{"1min":7.3819999999999997,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":18.181999999999999,"5min":42.523000000000003,"15min":42.523000000000003},"last":14.212},{"interface":"front","average":{"1min":14.087,"5min":15.260999999999999,"15min":17.285},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":21.725999999999999,"5min":39.066000000000003,"15min":39.066000000000003},"last":6.7569999999999997}]},{"osd":4,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.579000000000001,"5min":13.173,"15min":14.601000000000001},"min":{"1min":9.1319999999999997,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":34.061,"5min":34.061,"15min":34.061},"last":22.126999999999999},{"interface":"front","average":{"1min":19.207999999999998,"5min":15.503,"15min":15.571},"min":{"1min":11.127000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":32.462000000000003,"5min":32.462000000000003,"15min":32.462000000000003},"last":21.379999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.001999999999999,"5min":16.791,"15min":24.204999999999998},"min":{"1min":8.8490000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":24.170999999999999,"5min":190.821,"15min":190.821},"last":20.484999999999999},{"interface":"front","average":{"1min":17.789999999999999,"5min":18.347000000000001,"15min":24.542999999999999},"min":{"1min":6.1950000000000003,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":44.625999999999998,"5min":191.86199999999999,"15min":191.86199999999999},"last":25.943999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":20.004000000000001,"5min":18.93,"15min":24.388999999999999},"min":{"1min":11.297000000000001,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":33.362000000000002,"5min":166.45400000000001,"15min":166.45400000000001},"last":18.919},{"interface":"front","average":{"1min":18.824999999999999,"5min":19.602,"15min":24.550000000000001},"min":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.606000000000002,"5min":117.58499999999999,"15min":117.58499999999999},"last":25.146999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":23.681000000000001,"5min":23.681000000000001,"15min":23.681000000000001},"min":{"1min":12.381,"5min":12.381,"15min":12.381},"max":{"1min":52.109000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":22.834},{"interface":"front","average":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"min":{"1min":11.01,"5min":11.01,"15min":11.01},"max":{"1min":56.914000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":28.093}]}]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":266124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.0969999999999995,"15min":8.1890000000000001},"min":{"1min":6.0819999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":25.431000000000001,"5min":38.481000000000002,"15min":38.481000000000002},"last":8.1489999999999991},{"interface":"front","average":{"1min":10.859,"5min":11.725,"15min":12.146000000000001},"min":{"1min":5.7930000000000001,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":26.995000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":6.9189999999999996}]},{"osd":1,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":13.384,"15min":14.712},"min":{"1min":9.0299999999999994,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":26.780000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":10.257},{"interface":"front","average":{"1min":16.044,"5min":16.731999999999999,"15min":18.222000000000001},"min":{"1min":9.6699999999999999,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":28.262,"5min":54.338000000000001,"15min":54.338000000000001},"last":11.132}]},{"osd":3,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":11.717000000000001,"15min":11.206},"min":{"1min":7.7359999999999998,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":21.158000000000001,"5min":27.295999999999999,"15min":27.295999999999999},"last":32.860999999999997},{"interface":"front","average":{"1min":11.641,"5min":11.231,"15min":12.781000000000001},"min":{"1min":6.468,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":19.015999999999998,"5min":28.669,"15min":28.669},"last":11.763999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.824,"5min":14.007,"15min":14.811},"min":{"1min":7.2889999999999997,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":20.074999999999999,"5min":54.32,"15min":54.32},"last":19.866},{"interface":"front","average":{"1min":15.898,"5min":16.734000000000002,"15min":17.497},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":26.274000000000001,"5min":51.950000000000003,"15min":51.950000000000003},"last":12.297000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.038,"15min":15.157},"min":{"1min":7.0330000000000004,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":48.563000000000002,"5min":56.491,"15min":56.491},"last":21.747},{"interface":"front","average":{"1min":16.576000000000001,"5min":16.488,"15min":19.004000000000001},"min":{"1min":9.5960000000000001,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":47.328000000000003,"5min":75.540999999999997,"15min":75.540999999999997},"last":22.581}]},{"osd":6,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.277999999999999,"15min":18.053000000000001},"min":{"1min":10.872999999999999,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":28.962,"5min":30.213999999999999,"15min":30.213999999999999},"last":13.906000000000001},{"interface":"front","average":{"1min":14.414,"5min":13.6,"15min":14.318},"min":{"1min":8.2479999999999993,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":24.893999999999998,"5min":26.312999999999999,"15min":26.312999999999999},"last":24.859999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:58 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":17.641999999999999,"15min":17.641999999999999},"min":{"1min":12.304,"5min":12.304,"15min":12.304},"max":{"1min":30.283000000000001,"5min":30.283000000000001,"15min":30.283000000000001},"last":20.814},{"interface":"front","average":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":19.783000000000001},"min":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":31.402999999999999,"5min":31.402999999999999,"15min":31.402999999999999},"last":23.962}]}]},{"osd":0,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":270541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.576000000000001,"5min":11.565,"15min":11.337999999999999},"min":{"1min":4.8289999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":52.207000000000001,"5min":65.808000000000007,"15min":65.808000000000007},"last":8.4589999999999996},{"interface":"front","average":{"1min":12.784000000000001,"5min":12.163,"15min":11.606999999999999},"min":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":53.106999999999999,"5min":53.106999999999999,"15min":53.106999999999999},"last":13.702}]},{"osd":2,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.065,"15min":13.079000000000001},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":41.301000000000002,"5min":49.652999999999999,"15min":49.652999999999999},"last":17.263000000000002},{"interface":"front","average":{"1min":12.539,"5min":13.265000000000001,"15min":14.048},"min":{"1min":7.9960000000000004,"5min":6.798,"15min":6.798},"max":{"1min":21.843,"5min":47.484000000000002,"15min":47.484000000000002},"last":9.2100000000000009}]},{"osd":3,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.029,"5min":13.706,"15min":20.071999999999999},"min":{"1min":6.7320000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":32.063000000000002,"5min":205.096,"15min":205.096},"last":11.638999999999999},{"interface":"front","average":{"1min":18.690000000000001,"5min":17.489000000000001,"15min":22.172000000000001},"min":{"1min":8.5660000000000007,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":45.463000000000001,"5min":182.74000000000001,"15min":182.74000000000001},"last":16.628}]},{"osd":4,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.943,"5min":13.228,"15min":13.978},"min":{"1min":8.0660000000000007,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":29.585999999999999,"5min":45.112000000000002,"15min":45.112000000000002},"last":31.451000000000001},{"interface":"front","average":{"1min":17.719000000000001,"5min":16.870999999999999,"15min":16.803000000000001},"min":{"1min":7.1470000000000002,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":34.533000000000001,"5min":52.655999999999999,"15min":52.655999999999999},"last":30.16}]},{"osd":5,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":15.237,"5min":15.542,"15min":15.226000000000001},"min":{"1min":9.2129999999999992,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":25.379999999999999,"5min":43.088000000000001,"15min":43.088000000000001},"last":10.427},{"interface":"front","average":{"1min":16.452999999999999,"5min":16.266999999999999,"15min":16.77},"min":{"1min":9.7230000000000008,"5min":8.266,"15min":8.266},"max":{"1min":31.550000000000001,"5min":45.722000000000001,"15min":45.722000000000001},"last":7.7709999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":21.395,"5min":19.311,"15min":20.494},"min":{"1min":9.2309999999999999,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":44.292999999999999,"5min":78.338999999999999,"15min":78.338999999999999},"last":27.809000000000001},{"interface":"front","average":{"1min":18.216000000000001,"5min":17.864999999999998,"15min":20.274000000000001},"min":{"1min":10.634,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":42.417999999999999,"5min":80.850999999999999,"15min":80.850999999999999},"last":15.49}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":18.908999999999999,"5min":18.908999999999999,"15min":18.908999999999999},"min":{"1min":8.2469999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":61.445999999999998,"5min":61.445999999999998,"15min":61.445999999999998},"last":12.217000000000001},{"interface":"front","average":{"1min":20.067,"5min":20.067,"15min":20.067},"min":{"1min":11.833,"5min":11.833,"15min":11.833},"max":{"1min":63.518000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":14.566000000000001}]}]},{"osd":7,"up_from":60,"seq":257698037779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28716,"kb_used_data":732,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343124,"statfs":{"total":96636764160,"available":96607358976,"internally_reserved":0,"allocated":749568,"data_stored":279375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":11.984999999999999,"5min":11.984999999999999,"15min":11.984999999999999},"min":{"1min":4.452,"5min":4.452,"15min":4.452},"max":{"1min":55.622,"5min":55.622,"15min":55.622},"last":8.6039999999999992},{"interface":"front","average":{"1min":12.089,"5min":12.089,"15min":12.089},"min":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":31.178000000000001,"5min":31.178000000000001,"15min":31.178000000000001},"last":60.448999999999998}]},{"osd":1,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":48.959000000000003,"5min":48.959000000000003,"15min":48.959000000000003},"last":7.476},{"interface":"front","average":{"1min":13.994999999999999,"5min":13.994999999999999,"15min":13.994999999999999},"min":{"1min":8.0380000000000003,"5min":8.0380000000000003,"15min":8.0380000000000003},"max":{"1min":38.232999999999997,"5min":38.232999999999997,"15min":38.232999999999997},"last":11.103999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":12.195,"5min":12.195,"15min":12.195},"min":{"1min":5.952,"5min":5.952,"15min":5.952},"max":{"1min":20.297999999999998,"5min":20.297999999999998,"15min":20.297999999999998},"last":12.178000000000001},{"interface":"front","average":{"1min":15.340999999999999,"5min":15.340999999999999,"15min":15.340999999999999},"min":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":48.240000000000002,"5min":48.240000000000002,"15min":48.240000000000002},"last":15.457000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":13.568,"15min":13.568},"min":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"max":{"1min":54.603999999999999,"5min":54.603999999999999,"15min":54.603999999999999},"last":6.5570000000000004},{"interface":"front","average":{"1min":21.209,"5min":21.209,"15min":21.209},"min":{"1min":12.032999999999999,"5min":12.032999999999999,"15min":12.032999999999999},"max":{"1min":52.719000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":12.816000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.044,"5min":15.044,"15min":15.044},"min":{"1min":7.2329999999999997,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":32.390999999999998,"5min":32.390999999999998,"15min":32.390999999999998},"last":16.152999999999999},{"interface":"front","average":{"1min":13.667999999999999,"5min":13.667999999999999,"15min":13.667999999999999},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":9.2129999999999992}]},{"osd":5,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.632999999999999,"5min":19.632999999999999,"15min":19.632999999999999},"min":{"1min":12.234,"5min":12.234,"15min":12.234},"max":{"1min":52.197000000000003,"5min":52.197000000000003,"15min":52.197000000000003},"last":13.566000000000001},{"interface":"front","average":{"1min":17.359999999999999,"5min":17.359999999999999,"15min":17.359999999999999},"min":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":38.877000000000002,"5min":38.877000000000002,"15min":38.877000000000002},"last":17.347000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.231000000000002,"5min":19.231000000000002,"15min":19.231000000000002},"min":{"1min":12.494999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":39.508000000000003,"5min":39.508000000000003,"15min":39.508000000000003},"last":18.236000000000001},{"interface":"front","average":{"1min":14.912000000000001,"5min":14.912000000000001,"15min":14.912000000000001},"min":{"1min":8.6530000000000005,"5min":8.6530000000000005,"15min":8.6530000000000005},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":10.398999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542211,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":647168,"data_stored":231755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":8.9559999999999995,"5min":10.226000000000001,"15min":11.521000000000001},"min":{"1min":4.3490000000000002,"5min":3.484,"15min":3.484},"max":{"1min":16.178999999999998,"5min":45.506,"15min":45.506},"last":7.0860000000000003},{"interface":"front","average":{"1min":9.0839999999999996,"5min":10.619999999999999,"15min":12.609999999999999},"min":{"1min":5.1120000000000001,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":16.661000000000001,"5min":46.450000000000003,"15min":46.450000000000003},"last":9.9719999999999995}]},{"osd":1,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":11.772,"5min":13.038,"15min":15.023999999999999},"min":{"1min":7.4740000000000002,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":20.497,"5min":55.398000000000003,"15min":55.398000000000003},"last":20.712},{"interface":"front","average":{"1min":13.172000000000001,"5min":13.750999999999999,"15min":14.923},"min":{"1min":8.9809999999999999,"5min":6.569,"15min":6.569},"max":{"1min":22.056999999999999,"5min":41.811999999999998,"15min":41.811999999999998},"last":23.721}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":15.488,"15min":20.422000000000001},"min":{"1min":11.131,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":23.66,"5min":208.93799999999999,"15min":208.93799999999999},"last":8.8059999999999992},{"interface":"front","average":{"1min":13.587,"5min":16.247,"15min":24.045999999999999},"min":{"1min":7.3310000000000004,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":24.765000000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":12.638}]},{"osd":3,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":17.373999999999999,"5min":15.544,"15min":17.196999999999999},"min":{"1min":10.898999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":25.806999999999999,"5min":43.276000000000003,"15min":43.276000000000003},"last":25.545000000000002},{"interface":"front","average":{"1min":9.9079999999999995,"5min":10.916,"15min":14.124000000000001},"min":{"1min":6.4320000000000004,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":17.137,"5min":41.493000000000002,"15min":41.493000000000002},"last":8.0150000000000006}]},{"osd":4,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":16.132000000000001,"5min":16.742999999999999,"15min":22.754999999999999},"min":{"1min":9.4000000000000004,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":27.443999999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":27.823},{"interface":"front","average":{"1min":18.768000000000001,"5min":18.789000000000001,"15min":25.478000000000002},"min":{"1min":8.4570000000000007,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":29.385000000000002,"5min":111.874,"15min":111.874},"last":24.343}]},{"osd":5,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":12.522,"5min":12.273,"15min":16.116},"min":{"1min":7.3929999999999998,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":22.010000000000002,"5min":39.886000000000003,"15min":39.886000000000003},"last":9.3949999999999996},{"interface":"front","average":{"1min":17.369,"5min":17.016999999999999,"15min":21.588999999999999},"min":{"1min":11.013999999999999,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":30.199999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":11.706}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":20.678999999999998,"5min":20.678999999999998,"15min":20.678999999999998},"min":{"1min":13.064,"5min":13.064,"15min":13.064},"max":{"1min":31.757999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":26.198},{"interface":"front","average":{"1min":15.526999999999999,"5min":15.526999999999999,"15min":15.526999999999999},"min":{"1min":9.6890000000000001,"5min":9.6890000000000001,"15min":9.6890000000000001},"max":{"1min":28.809000000000001,"5min":28.809000000000001,"15min":28.809000000000001},"last":21.055}]}]},{"osd":5,"up_from":14,"seq":60129542211,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":679936,"data_stored":629595,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":9.0600000000000005,"5min":10.461,"15min":13.067},"min":{"1min":4.5670000000000002,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":18.443000000000001,"5min":45.491,"15min":45.491},"last":7.9829999999999997},{"interface":"front","average":{"1min":8.7940000000000005,"5min":10.765000000000001,"15min":12.897},"min":{"1min":4.4349999999999996,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":14.475,"5min":49.991999999999997,"15min":49.991999999999997},"last":5.851}]},{"osd":1,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":11.146000000000001,"5min":13.614000000000001,"15min":21.236000000000001},"min":{"1min":5.6900000000000004,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":18.754000000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":9.1950000000000003},{"interface":"front","average":{"1min":10.789,"5min":17.363,"15min":29.614999999999998},"min":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":17.491,"5min":344.79599999999999,"15min":344.79599999999999},"last":6.6589999999999998}]},{"osd":2,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.129,"5min":12.884,"15min":13.379},"min":{"1min":9.1910000000000007,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":24.969999999999999,"5min":53.235999999999997,"15min":53.235999999999997},"last":14.609},{"interface":"front","average":{"1min":14.458,"5min":13.545999999999999,"15min":15.108000000000001},"min":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":26.155000000000001,"5min":54.747,"15min":54.747},"last":13.794}]},{"osd":3,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.762,"5min":14.917,"15min":13.975},"min":{"1min":9.6069999999999993,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":24.18,"5min":55.310000000000002,"15min":55.310000000000002},"last":11.731},{"interface":"front","average":{"1min":11.239000000000001,"5min":12.801,"15min":16.099},"min":{"1min":6.9889999999999999,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":19.765000000000001,"5min":51.183,"15min":51.183},"last":11.435}]},{"osd":4,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":12.808,"5min":14.959,"15min":19.792999999999999},"min":{"1min":9.0570000000000004,"5min":6.085,"15min":6.085},"max":{"1min":21.306999999999999,"5min":281.608,"15min":281.608},"last":12.446},{"interface":"front","average":{"1min":16.542999999999999,"5min":16.850999999999999,"15min":21.760999999999999},"min":{"1min":11.683999999999999,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":23.295999999999999,"5min":225.83600000000001,"15min":225.83600000000001},"last":15.252000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":16.638999999999999,"5min":16.809000000000001,"15min":20.645},"min":{"1min":11.105,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":22.853000000000002,"5min":198.667,"15min":198.667},"last":13.159000000000001},{"interface":"front","average":{"1min":16.501000000000001,"5min":17.471,"15min":22.067},"min":{"1min":10.128,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":24.225000000000001,"5min":208.446,"15min":208.446},"last":16.010999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":14.349,"5min":14.349,"15min":14.349},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":10.851000000000001},{"interface":"front","average":{"1min":18.501000000000001,"5min":18.501000000000001,"15min":18.501000000000001},"min":{"1min":12.881,"5min":12.881,"15min":12.881},"max":{"1min":25.683,"5min":25.683,"15min":25.683},"last":30.606999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542211,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":677215,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.8759999999999994,"5min":8.9649999999999999,"15min":10.57},"min":{"1min":5.6459999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":26.015000000000001,"5min":40.225999999999999,"15min":40.225999999999999},"last":9.7699999999999996},{"interface":"front","average":{"1min":11.209,"5min":10.295999999999999,"15min":11.488},"min":{"1min":5.8639999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":20.613,"5min":30.405000000000001,"15min":30.405000000000001},"last":10.917999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":10.425000000000001,"5min":13.679,"15min":11.686},"min":{"1min":7.0549999999999997,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":21.376000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":30.559000000000001},{"interface":"front","average":{"1min":15.489000000000001,"5min":17.225999999999999,"15min":16.045999999999999},"min":{"1min":10.004,"5min":6.415,"15min":6.415},"max":{"1min":30.391999999999999,"5min":164.911,"15min":164.911},"last":31.259}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.678000000000001,"5min":14.433999999999999,"15min":19.654},"min":{"1min":5.359,"5min":5.359,"15min":5.359},"max":{"1min":20.265999999999998,"5min":122.762,"15min":122.762},"last":15.545},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.653,"15min":19.021000000000001},"min":{"1min":7.1059999999999999,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":22.109000000000002,"5min":129.99100000000001,"15min":129.99100000000001},"last":6.4720000000000004}]},{"osd":4,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.468999999999999,"5min":18.315000000000001,"15min":27.951000000000001},"min":{"1min":9.1140000000000008,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":49.097000000000001,"5min":359.46300000000002,"15min":359.46300000000002},"last":11.789},{"interface":"front","average":{"1min":15.288,"5min":17.658999999999999,"15min":26.722999999999999},"min":{"1min":6.4539999999999997,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":47.192,"5min":314.47699999999998,"15min":314.47699999999998},"last":13.113}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":19.167999999999999,"5min":21.059999999999999,"15min":25.846},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":81.923000000000002,"5min":152.97399999999999,"15min":152.97399999999999},"last":27.785},{"interface":"front","average":{"1min":16.448,"5min":18.698,"15min":24.085000000000001},"min":{"1min":10.108000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.151000000000003,"5min":154.53100000000001,"15min":154.53100000000001},"last":17.027000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.266999999999999,"5min":15.220000000000001,"15min":17.032},"min":{"1min":8.218,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":20.634,"5min":43.045000000000002,"15min":43.045000000000002},"last":26.972999999999999},{"interface":"front","average":{"1min":16.388000000000002,"5min":17.977,"15min":20.286999999999999},"min":{"1min":11.009,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":27.966999999999999,"5min":41.356000000000002,"15min":41.356000000000002},"last":14.042999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":19.687999999999999,"15min":19.687999999999999},"min":{"1min":11.547000000000001,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":32.191000000000003,"5min":32.191000000000003,"15min":32.191000000000003},"last":17.984000000000002},{"interface":"front","average":{"1min":19.468,"5min":19.468,"15min":19.468},"min":{"1min":13.093,"5min":13.093,"15min":13.093},"max":{"1min":36.454000000000001,"5min":36.454000000000001,"15min":36.454000000000001},"last":29.050000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":647168,"data_stored":231755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5269999999999992,"5min":11.606,"15min":13.595000000000001},"min":{"1min":4.9740000000000002,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":23.562000000000001,"5min":137.13999999999999,"15min":137.13999999999999},"last":6.4900000000000002},{"interface":"front","average":{"1min":9.0630000000000006,"5min":12.724,"15min":17.661999999999999},"min":{"1min":4.1959999999999997,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":23.231000000000002,"5min":199.268,"15min":199.268},"last":15.430999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.608000000000001,"15min":18.324000000000002},"min":{"1min":5.2779999999999996,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":25.949999999999999,"5min":92.817999999999998,"15min":92.817999999999998},"last":9.4589999999999996},{"interface":"front","average":{"1min":11.914,"5min":14.56,"15min":19.475999999999999},"min":{"1min":7.8040000000000003,"5min":5.556,"15min":5.556},"max":{"1min":17.295000000000002,"5min":95.665999999999997,"15min":95.665999999999997},"last":10.676}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":13.292999999999999,"5min":17.664000000000001,"15min":23.103999999999999},"min":{"1min":7.4470000000000001,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":23.303999999999998,"5min":208.06,"15min":208.06},"last":18.091999999999999},{"interface":"front","average":{"1min":13.791,"5min":16.366,"15min":18.635999999999999},"min":{"1min":8.7569999999999997,"5min":6.915,"15min":6.915},"max":{"1min":21.555,"5min":147.18199999999999,"15min":147.18199999999999},"last":18.949999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":13.999000000000001,"15min":18.155000000000001},"min":{"1min":6.2610000000000001,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":28.175999999999998,"5min":121.627,"15min":121.627},"last":10.128},{"interface":"front","average":{"1min":15.657999999999999,"5min":16.553999999999998,"15min":18.242000000000001},"min":{"1min":10.692,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":25.760999999999999,"5min":74.602000000000004,"15min":74.602000000000004},"last":14.593999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":16.504000000000001,"5min":16.984000000000002,"15min":18.885000000000002},"min":{"1min":8.3930000000000007,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":24.431000000000001,"5min":71.759,"15min":71.759},"last":11.355},{"interface":"front","average":{"1min":13.960000000000001,"5min":14.284000000000001,"15min":16.748000000000001},"min":{"1min":6.6399999999999997,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":29.401,"5min":70.882999999999996,"15min":70.882999999999996},"last":12.106999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.581,"5min":16.417999999999999,"15min":19.870000000000001},"min":{"1min":8.9700000000000006,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":22.050999999999998,"5min":80.668999999999997,"15min":80.668999999999997},"last":13.053000000000001},{"interface":"front","average":{"1min":17.324999999999999,"5min":19.062999999999999,"15min":21.738},"min":{"1min":12.243,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":25.762,"5min":72.796999999999997,"15min":72.796999999999997},"last":16.177}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":17.497,"5min":17.497,"15min":17.497},"min":{"1min":11.442,"5min":11.442,"15min":11.442},"max":{"1min":29.594000000000001,"5min":29.594000000000001,"15min":29.594000000000001},"last":17.23},{"interface":"front","average":{"1min":18.670999999999999,"5min":18.670999999999999,"15min":18.670999999999999},"min":{"1min":11.663,"5min":11.663,"15min":11.663},"max":{"1min":27.414000000000001,"5min":27.414000000000001,"15min":27.414000000000001},"last":19.895}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:27:20.918 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [2, 5] 2023-12-06T09:27:20.919 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.4', '2', '5'] 2023-12-06T09:27:20.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.4 2 5 2023-12-06T09:27:20.976 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:20.976 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:20.976 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:20.977 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:20.977 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:20.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:21.027 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:21.041+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:21.100 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:21.114+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:21.131 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:21.131 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:21.131 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:21.131 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:21.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:21.201 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:21.215+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:21.302 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:21.316+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:21.312 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:21.313 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:21.313 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:21.322 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:21.403 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:21.417+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:21.454 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:21.454 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:21.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:21.455 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:27:21.455 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:21.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:21.503 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:21.517+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:21.605 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:21.619+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:21.640 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:21.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:21.706 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:21.720+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:21.796 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2023-12-06T09:27:21.828 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:21.828 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:21.828 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:21.828 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:21.828 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:21.828 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:21.841+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:21.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:21.839 INFO:teuthology.orchestra.run.smithi088.stderr:set 2.4 pg_upmap mapping to [2,5] 2023-12-06T09:27:21.898 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:27:21.907 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:21.921+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:22.008 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:22.022+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.13803233599998, 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.044 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:27:22.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.72032221900002, 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.716137774, 2023-12-06T09:27:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:22.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:27:22.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:27:22.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:27:22.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:27:22.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.061 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.630606688, 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.316630073, 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.067 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:22.072 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.31511222900002, 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:22.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:27:22.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:27:22.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:27:22.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.31444685399998, 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:27:22.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.31018668199999, 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:22.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:27:22.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:27:22.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:27:22.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:27:22.091 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.30955347299999, 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.092 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:27:22.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.097 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.30892637099998, 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:27:22.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:27:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:27:22.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:27:22.102 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.22066659299998, 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.103 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 322.21974561299999, 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 321.46736768199997, 2023-12-06T09:27:22.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:27:22.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:27:22.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:27:22.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.59733271099998, 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:27:22.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:27:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:27:22.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:27:22.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.59615919999999, 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.124 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.58800722900003, 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:27:22.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:27:22.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.57772529300001, 2023-12-06T09:27:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:27:22.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.57706741300001, 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:22.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:27:22.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:27:22.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.145 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.576392634, 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 313.54403504300001, 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:22.151 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:27:22.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:27:22.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:27:22.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.156 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:22.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:22.159 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:22.172+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:22.259 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:22.273+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:22.329 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:22.329 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:22.329 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:22.329 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:22.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:22.360 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:22.374+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:22.461 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:22.474+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:22.467 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:22.468 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:22.468 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:22.468 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:22.468 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:22.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:22.562 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:22.576+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:22.610 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 171.971701962, 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:22.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:22.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 171.66648066100001, 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:22.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:22.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 6.7827428689999998, 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029206599999999998, 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:22.618 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 12 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:22.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835047+0000", 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5968999999999999e-05 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835217+0000", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017016899999999999 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835416+0000", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000198677 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835585+0000", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016903299999999999 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:22.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.836489+0000", 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00090463399999999997 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.837952+0000", 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001462178 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:22.622 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:22.623 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:22.662 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:22.676+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:22.721 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:22.763 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:22.777+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:22.770 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:22.770 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:22.770 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:22.771 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:22.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:22.813 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2023-12-06T09:27:22.864 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:22.878+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:22.933 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:22.933 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:22.933 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:22.933 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:22.933 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:22.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:22.965 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:22.979+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 308.13757562699999, 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:23.099 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:23.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:23.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:23.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:23.103 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:23.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:23.109 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:23.109 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:23.109 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:23.122+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:23.167 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:23.181+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:23.193 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:23.193 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:23.239 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:23.240 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:23.240 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:23.240 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:23.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:23.290 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:23.294 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:23.308+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:23.369 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:23.383+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:23.385 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:23.385 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:23.385 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:23.385 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:23.385 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:23.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:23.395 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:23.396 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:23.470 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:23.484+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:23.485 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:23.485 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:23.571 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:23.585+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:23.595 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:23.595 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:23.601 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:23.604 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:23.672 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:23.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:23.688 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:23.689 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:23.733 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:23.733 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:23.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:23.773 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:23.787+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:23.777 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:23.777 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:23.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2023-12-06T09:27:23.896 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:23.910+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:23.912 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:23.912 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:23.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:27:23.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:23.975 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:23.989+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:24.055 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:24.055 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:24.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:24.107 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:24.121+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:24.177 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:24.191+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:24.198 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:24.199 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:24.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:24.291 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:24.305+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:24.374 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:24.375 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:24.379 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:24.393+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:24.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:24.480 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:24.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:24.554 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:24.554 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:24.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:24.581 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:24.596+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:24.682 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:24.696+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:24.690 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:24.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:24.786 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:24.800+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:24.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2023-12-06T09:27:24.875 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:24.876 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:24.885 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:24.886 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:27:24.886 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:24.900+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:24.985 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:24.999+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:25.056 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:25.056 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:25.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:25.110 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:25.124+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:25.188 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:25.202+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:25.195 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:25.196 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:25.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:25.289 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:25.303+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:25.385 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:25.385 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:25.390 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:25.404+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:25.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:25.491 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:25.506+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:25.526 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:25.526 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:25.536 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:25.592 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:25.606+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:25.665 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:25.665 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:25.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:25.693 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:25.707+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:25.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2023-12-06T09:27:25.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:25.824 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:25.824 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:25.825 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:25.839+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:25.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:25.923 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:25.937+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:25.996 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:26.010+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:26.012 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:26.013 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:26.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:26.038 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:26.038 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:26.100 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:26.114+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:26.128 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:26.128 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:26.201 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:26.215+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:26.227 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:26.227 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:26.231 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:26.231 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:26.231 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:26.231 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:26.243 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:26.328 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:26.342+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:26.352 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:26.352 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:26.403 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:26.417+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:26.430 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:26.431 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:26.431 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:26.431 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:26.431 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:26.437 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:26.437 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:26.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:26.525 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:26.538+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:26.542 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:26.614 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 175.97746574000001, 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:26.615 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:26.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:26.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 175.672244439, 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:26.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:26.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 10.788506647, 2023-12-06T09:27:26.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029206599999999998, 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 12 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:26.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835047+0000", 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5968999999999999e-05 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835217+0000", 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017016899999999999 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:26.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835416+0000", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000198677 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835585+0000", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016903299999999999 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.836489+0000", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00090463399999999997 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.837952+0000", 2023-12-06T09:27:26.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001462178 2023-12-06T09:27:26.626 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.626 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:26.626 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.626 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:26.626 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:26.626 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:26.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:26.628 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:26.629 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:26.629 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:26.642+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:26.707 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:26.721+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:26.714 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:26.769 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:26.769 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:26.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:26.809 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:26.823+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:26.860 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:27:26.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:27:26.860 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 2, 3, 4, 6, 7, 0, 5] 2023-12-06T09:27:26.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2023-12-06T09:27:26.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:27:26.912 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:26.913 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:26.913 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:26.913 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:26.913 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:26.913 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:26.927+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:26.924 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:26.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2023-12-06T09:27:27.037 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:27.051+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:27.112 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:27.126+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 312.17777034800002, 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:27.139 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:27.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:27.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:27.143 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:27.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:27.143 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:27.143 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:27.143 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:27.149 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:27.213 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:27.227+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:27.289 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:27.289 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:27.289 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:27.289 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:27.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:27.339 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:27.353+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:27.415 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:27.429+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:27.467 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:27.467 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:27.467 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:27.467 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:27.467 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:27.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:27.515 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:27.529+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:27.572 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:27:27.575+0000 ff5f640 -1 mon.a@0(leader).osd e88 definitely_dead 0 2023-12-06T09:27:27.607 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:27.607 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:27.607 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:27.607 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:27.607 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:27.616 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:27.616 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:27.630+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:27.717 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:27.731+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:27.743 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:27.743 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:27.743 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:27.743 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:27.752 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:27.818 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:27.832+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:27.892 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:27.902 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:27.902 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:27.902 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:27.902 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:27.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:27.919 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:05.897 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:27.921 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:05.900 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:27.937 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:05.916 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:27.947 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:27.961+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:27.957 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:00.271 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:27.959 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:00.273 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:27.981 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:00.295 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:28.020 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:28.033+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:28.077 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:28.077 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:28.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 329.37040914300002, 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:27:28.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:27:28.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:27:28.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.084 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.67595527399999, 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:27:28.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:27:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:27:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.23306279799999, 2023-12-06T09:27:28.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.22423651700001, 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:27:28.094 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.098 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.22361484499999, 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.21566836400001, 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:27:28.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:27:28.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.106 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.21519434599998, 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:28.107 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:27:28.108 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:27:28.109 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.110 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 328.21473999099999, 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.111 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.112 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:27:28.113 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.114 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 327.83074773999999, 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:28.115 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:27:28.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.117 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:28.118 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 327.445598335, 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.119 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.120 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.121 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 327.433323024, 2023-12-06T09:27:28.122 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.123 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.124 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:27:28.125 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.126 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 327.43183744599997, 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.127 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.128 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.129 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.52530028799998, 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.131 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.132 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.133 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.520167985, 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.135 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.136 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.137 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.138 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:27:28.139 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.51934195500002, 2023-12-06T09:27:28.141 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:27:28.141 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.141 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.141 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:27:28.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.144 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:27:28.145 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.51677249400001, 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.146 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.147 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.148 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.149 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.51602702399998, 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.150 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.151 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:27:28.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:27:28.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.154 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.51522095299998, 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.156 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.157 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:28.158 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 319.504116501, 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.159 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.160 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:27:28.161 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.162 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 170.73078436399999, 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.163 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.164 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.165 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.166 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:28.168 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:28.168 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:28.182+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:28.214 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:27:28.268 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:28.282+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:28.291 INFO:teuthology.orchestra.run.smithi088.stderr:marked out osd.4. 2023-12-06T09:27:28.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2023-12-06T09:27:28.349 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:28.349 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:28.349 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:28.349 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:28.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:28.403 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:28.417+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:28.471 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:28.485+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:28.526 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:28.526 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:28.526 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:28.526 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:28.526 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:28.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:28.572 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:28.586+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:28.673 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:28.686+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:28.705 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.705 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 13.496424838999999, 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010005948000000001, 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 11 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:28.706 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221310+0000", 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6829000000000001e-05 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221505+0000", 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019430599999999999 2023-12-06T09:27:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221716+0000", 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021168399999999999 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.224250+0000", 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025338729999999999 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.229715+0000", 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054648589999999999 2023-12-06T09:27:28.708 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.231300+0000", 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001584397 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:28.709 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:28.714 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:28.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:28.774 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:28.787+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:28.848 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:28.848 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:28.849 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:28.849 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:28.858 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:28.902 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:28.916+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:28.976 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:28.990+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:28.983 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:28.984 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:28.984 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:28.985 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:28.985 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:28.985 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:28.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:29.072 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:29.072 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:29.077 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:29.090+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:29.156 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:29.163 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:29.163 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:29.163 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:29.163 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:29.163 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:29.172 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:29.216 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:29.229+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:29.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2023-12-06T09:27:29.246 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:29.247 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:29.290 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:29.304+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:29.344 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:29.344 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:29.355 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:29.416 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:29.430+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:29.417 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:29.417 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:29.481 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:29.494+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:29.521 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:29.521 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:29.525 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:29.525 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:29.536 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:29.621 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:29.634+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:29.623 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:29.623 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:29.683 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:29.697+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:29.711 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:29.711 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:29.720 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:29.720 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:29.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:29.815 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:29.828+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:29.885 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:29.899+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:29.902 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:29.902 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:29.912 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:30.002 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:30.015+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:30.083 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:30.083 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:30.087 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:30.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:30.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:30.188 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:30.202+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:30.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:27:30.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2023-12-06T09:27:30.279 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:30.280 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:30.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:30.292 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:30.306+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:30.399 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:30.408+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:30.487 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:30.488 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:30.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:30.501 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:30.515+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:30.593 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:30.607+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:30.630 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:30.630 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:30.640 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:30.724 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:30.738+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:30.801 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:30.814+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:30.814 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:30.814 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:30.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:30.909 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:30.923+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:30.995 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:30.995 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:30.998 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:31.012+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:31.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:31.109 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:31.121+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:31.177 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:31.177 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:31.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:31.200 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:31.214+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:31.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2023-12-06T09:27:31.302 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:31.315+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:27:31.365 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:31.365 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:31.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:31.420 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:31.434+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:31.503 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:31.517+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:31.527 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:31.527 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:31.536 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:31.621 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:31.635+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:31.704 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:31.705 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:31.705 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:31.719+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:31.714 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:31.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:31.806 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:31.820+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:31.897 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:31.897 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:31.908 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:31.908 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:31.921+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:31.990 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:32.009 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:32.023+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:32.076 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:32.076 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:32.076 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:32.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:32.110 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:32.124+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:32.159 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:32.210 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:32.224+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:32.212 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:32.212 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:32.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:32.245 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:32.245 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:32.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2023-12-06T09:27:32.333 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:32.333 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:32.347+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:32.365 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:32.365 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:32.374 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:32.418 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:32.432+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:32.432 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:32.513 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:32.527+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:32.532 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:32.532 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:32.532 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:32.533 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:32.533 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:32.533 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:32.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:32.614 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:32.628+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:32.625 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:32.711 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:32.712 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:32.712 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:32.712 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:32.712 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:32.715 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:32.728+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:32.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:32.815 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:32.829+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 17.643374287, 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010005948000000001, 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:32.852 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 11 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:32.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221310+0000", 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6829000000000001e-05 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221505+0000", 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019430599999999999 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221716+0000", 2023-12-06T09:27:32.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021168399999999999 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.224250+0000", 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025338729999999999 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.229715+0000", 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054648589999999999 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.231300+0000", 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001584397 2023-12-06T09:27:32.855 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:32.856 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:32.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:32.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:32.856 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:32.856 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:32.861 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:32.916 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:32.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:32.993 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:32.993 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:32.993 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:32.993 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:33.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:33.047 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:33.061+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:33.119 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:33.133+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:33.132 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:33.133 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:33.133 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:33.133 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:33.133 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:33.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:33.226 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:33.240+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:33.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2023-12-06T09:27:33.306 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 7, 0, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:27:33.306 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:27:33.312 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:33.312 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:33.312 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:33.312 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:33.312 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:33.322 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:33.323 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:33.336+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:33.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:27:33.423 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:33.437+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:33.496 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:33.496 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:33.496 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:33.496 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:33.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:33.547 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:33.561+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:33.627 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:33.641+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:33.694 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:33.694 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:33.694 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:33.694 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:33.694 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:33.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:33.732 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:33.745+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:33.832 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:33.846+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:33.899 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:33.900 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:33.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:33.900 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:27:33.900 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:33.912 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:33.935 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:33.949+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:34.035 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:34.050+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:34.041 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:34.041 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:34.041 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:34.041 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:34.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:34.138 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:34.153+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:34.183 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:34.183 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:34.183 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:34.183 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:34.184 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:34.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:34.239 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:34.253+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:34.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2023-12-06T09:27:34.335 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:27:34.342 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:34.356+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:34.389 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:34.389 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:34.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:34.389 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:34.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 337.47651559299999, 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:27:34.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:27:34.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:27:34.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 335.05880547599997, 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.398 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:27:34.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:27:34.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:27:34.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:27:34.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:27:34.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.409 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 335.05462103100001, 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:27:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:27:34.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:27:34.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.415 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.96908994500001, 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.416 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:27:34.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:27:34.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:27:34.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.65511333000001, 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:27:34.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:27:34.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:27:34.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:27:34.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.65359548599997, 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:27:34.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:27:34.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:27:34.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.65293011099999, 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.433 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:27:34.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:27:34.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.648669939, 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.64803673, 2023-12-06T09:27:34.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:27:34.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:27:34.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.64740962799999, 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:27:34.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:27:34.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:27:34.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:27:34.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.55914984999998, 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:27:34.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:27:34.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:27:34.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 334.55822886999999, 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.460 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 333.80585093899998, 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.93581596799999, 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:27:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:27:34.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:27:34.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.934642457, 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.476 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:27:34.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:27:34.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.92649048599998, 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.481 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:27:34.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:27:34.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:27:34.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.486 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.91620855000002, 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:34.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:27:34.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:27:34.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:27:34.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.491 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.91555067000002, 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.492 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:27:34.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:27:34.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:34.497 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.91487589100001, 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:27:34.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:27:34.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:27:34.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 325.88251830000002, 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:27:34.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:27:34.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:27:34.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:27:34.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:34.508 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:34.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:34.510 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:34.523+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:34.611 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:34.624+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:34.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:34.680 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:34.680 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:34.680 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:34.680 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:34.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:34.712 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:34.724+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:34.811 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:34.825+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:34.865 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:34.865 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:34.865 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:34.865 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:34.865 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:34.875 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:34.876 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:34.876 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:34.913 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:34.927+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:35.003 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:35.003 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:35.013 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:35.028+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 184.37758673900001, 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:35.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:35.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 184.07236543799999, 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.019 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:35.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.022 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.022 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:35.022 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:35.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:35.022 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.022 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:35.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 19.188627646, 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029206599999999998, 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 12 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:35.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.026 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:35.026 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:35.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:35.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835047+0000", 2023-12-06T09:27:35.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5968999999999999e-05 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835217+0000", 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017016899999999999 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835416+0000", 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000198677 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835585+0000", 2023-12-06T09:27:35.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016903299999999999 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.836489+0000", 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00090463399999999997 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:35.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:35.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:35.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.837952+0000", 2023-12-06T09:27:35.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001462178 2023-12-06T09:27:35.031 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.031 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:35.031 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.032 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:35.032 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:35.032 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:35.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:35.104 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:35.114 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:35.128+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:35.211 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:35.211 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:35.211 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:35.211 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:35.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:35.238 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:35.251+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:35.297 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:35.298 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:35.316 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:35.330+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:35.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2023-12-06T09:27:35.384 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:35.384 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:35.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:35.445 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:35.459+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:35.468 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:35.468 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:35.519 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:35.533+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:35.551 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:35.590 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:35.590 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:35.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:35.644 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:35.658+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:35.723 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:35.737+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:35.732 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:35.732 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:35.741 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:35.824 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:35.838+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:35.923 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:35.923 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:35.925 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:35.939+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:35.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:36.045 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:36.059+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:36.059 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:36.059 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:36.069 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:36.127 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:36.141+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:36.228 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:36.243+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:36.237 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:36.237 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:36.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:36.330 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:36.344+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:36.350 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:27:36.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2023-12-06T09:27:36.423 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:36.423 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:36.431 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:36.457 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:36.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:36.532 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:36.546+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:36.561 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:36.561 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:36.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:36.633 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:36.648+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:36.701 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:36.702 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:36.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:36.735 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:36.749+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:36.835 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:36.849+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:36.879 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:36.879 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:36.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:36.936 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:36.951+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:37.018 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:37.018 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:37.029 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:37.070 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:37.085+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:37.138 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:37.152+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:37.196 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:37.197 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:37.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:37.238 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:37.252+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:37.332 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:37.332 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:37.340 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:37.354+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:37.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:37.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2023-12-06T09:27:37.441 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:37.456+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:37.478 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:37.478 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:37.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:37.541 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:27:37.546 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:37.560+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:37.564 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:37.643 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:37.657+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:37.663 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:37.663 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:37.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:37.744 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:37.759+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:37.845 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:37.860+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:37.858 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:37.859 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:37.913 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:37.913 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:37.913 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:37.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:37.978 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:37.978 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:37.979 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:37.992+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:38.046 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:38.060+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:38.052 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:38.053 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:38.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:38.053 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:27:38.053 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:38.063 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:38.072 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:38.073 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:38.147 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:38.161+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:38.166 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:38.166 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:38.210 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:38.211 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:38.211 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:38.211 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:38.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:38.250 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:38.265+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:38.251 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:38.251 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:38.307 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 7, 0, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:27:38.308 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:27:38.336 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:38.336 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:38.349 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:38.363+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:38.381 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:38.381 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:38.381 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:38.381 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:38.381 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:38.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:38.419 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:38.420 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:38.463 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:38.477+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:38.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2023-12-06T09:27:38.506 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:38.506 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:38.553 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:38.567+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:27:38.628 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 341.71735854399998, 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:27:38.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.635 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 339.29964842700002, 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:27:38.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 339.295463982, 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:38.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:27:38.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:27:38.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 339.209932896, 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:27:38.648 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:38.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:27:38.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.653 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.895956281, 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.654 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.89443843700002, 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:27:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:27:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:27:38.667 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.89377306199998, 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:27:38.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:27:38.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:27:38.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.88951288999999, 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.673 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.88887968099999, 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.679 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.681 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:27:38.682 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:27:38.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.88825257899998, 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.687 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.688 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.79999280099997, 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:27:38.689 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:27:38.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.79907182099998, 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:38.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:27:38.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 338.04669388999997, 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.701 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:27:38.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:27:38.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:27:38.706 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.17665891899998, 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:27:38.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:27:38.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.17548540799999, 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.712 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:27:38.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.717 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.16733343700002, 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.718 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.15705150100001, 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.724 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:27:38.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:27:38.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.15639362100001, 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.729 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:27:38.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:27:38.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:27:38.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.155718842, 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.735 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.738 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 330.12336125100001, 2023-12-06T09:27:38.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:38.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:38.745 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:38.746 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:38.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:38.748 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:38.762+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:38.848 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:38.862+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:38.920 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:38.920 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:38.920 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:38.920 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:38.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:38.950 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:38.964+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:39.051 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:39.065+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:39.061 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:39.061 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:39.061 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:39.061 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:39.061 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:39.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:39.155 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:39.169+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:39.247 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 188.609802989, 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:39.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:39.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 188.30458168800001, 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:39.251 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:39.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:39.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:39.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 23.420843896000001, 2023-12-06T09:27:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029206599999999998, 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 12 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:39.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835047+0000", 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5968999999999999e-05 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835217+0000", 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017016899999999999 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835416+0000", 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000198677 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835585+0000", 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016903299999999999 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.836489+0000", 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00090463399999999997 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.837952+0000", 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001462178 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:39.259 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:39.260 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:39.261 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:39.275+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:39.362 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:39.376+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:39.430 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:39.430 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:39.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:39.484 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:39.498+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:39.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2023-12-06T09:27:39.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:27:39.563 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:39.577+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:39.571 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:39.571 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:39.572 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:39.572 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:39.572 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:39.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:39.666 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:39.680+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:39.761 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:39.761 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 324.79884293499998, 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:39.762 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:39.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:39.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.765 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:39.766 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.766 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:39.766 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:39.766 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:39.767 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:39.781+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:39.769 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:39.868 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:39.882+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:39.898 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:39.898 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:39.898 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:39.898 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:39.907 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:39.969 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:39.983+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:40.070 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:40.084+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:40.076 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:40.076 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:40.076 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:40.076 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:40.076 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:40.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:40.171 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:40.185+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:40.214 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:40.214 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:40.214 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:40.214 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:40.214 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:40.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:40.272 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:40.286+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:40.373 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:40.387+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:40.391 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:40.391 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:40.391 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:40.391 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:40.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:40.474 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:40.488+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:40.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:40.546 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:40.546 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:40.546 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:40.546 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:40.546 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:40.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:27:40.575 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:40.589+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:40.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2023-12-06T09:27:40.680 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:40.694+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:40.767 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 342.06213558000002, 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:40.768 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.769 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:27:40.770 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.771 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 341.36768171099999, 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:27:40.772 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.774 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:27:40.775 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.776 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:27:40.778 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.92478923499999, 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.779 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:27:40.780 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:27:40.781 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:27:40.782 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:27:40.783 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.783 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.784 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.91596295400001, 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:40.785 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.786 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:27:40.787 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.788 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.789 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.91534128199999, 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.791 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.792 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:27:40.793 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:27:40.793 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:27:40.794 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.90739480100001, 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.795 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:27:40.796 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.798 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.799 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.90692078299998, 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:40.800 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.801 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.802 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:27:40.803 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.804 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.90646642799999, 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.805 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.806 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:27:40.807 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:27:40.808 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.52247417699999, 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:27:40.809 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.810 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.811 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.812 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.137324772, 2023-12-06T09:27:40.813 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:40.814 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:27:40.815 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.816 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.817 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.125049461, 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.818 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.819 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.821 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 340.12356388299997, 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.822 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:27:40.823 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.825 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:27:40.826 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.21702672499998, 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:40.827 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:27:40.828 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:27:40.829 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:27:40.830 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.211894422, 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.831 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.832 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.833 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.834 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.835 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.21106839200002, 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.836 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.837 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.839 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.20849893100001, 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.840 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.841 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.842 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.843 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.20775346099998, 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.844 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.845 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:27:40.846 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:27:40.847 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.20694738999998, 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:27:40.848 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.849 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:27:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:27:40.851 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.852 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 332.195842938, 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.853 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.854 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.856 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 183.42251080099999, 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.857 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:27:40.858 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:27:40.859 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:40.860 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:40.861 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:40.862 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:40.863 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:40.863 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:40.876+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:40.882 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:40.963 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:40.977+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:40.968 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:40.969 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:41.033 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:41.033 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:41.034 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:41.034 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:41.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:41.056 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:41.065 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:41.078+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:41.150 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:41.150 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:41.166 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:41.179+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:41.190 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:41.190 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:41.191 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:41.191 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:41.191 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:41.199 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:41.247 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:41.247 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:41.284 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:41.298+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:41.334 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:41.334 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:41.367 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:41.381+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:41.371 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:41.372 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:41.380 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:41.421 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:41.469 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:41.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:41.557 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:41.557 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:41.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:41.590 INFO:tasks.mon_recovery:adding mon c back in 2023-12-06T09:27:41.591 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T09:27:41.591 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T09:27:41.593 INFO:tasks.ceph.mon.c:Started 2023-12-06T09:27:41.593 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T09:27:41.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:27:41.608 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:41.622+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:41.671 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:41.685+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:41.736 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:41.737 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:41.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:41.772 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:41.786+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:41.875 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:41.889+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:41.876 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:41.876 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:41.885 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:41.975 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:41.988+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:42.040 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:42.040 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:42.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:42.095 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:42.109+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:42.190 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:42.202+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:42.210 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:42.211 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:42.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:42.277 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:42.291+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:42.282 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":35,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:27:42.378 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:42.392+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:42.391 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:42.392 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:42.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:42.479 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:42.493+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:42.531 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:42.531 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:42.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:42.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:27:42.582 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:42.596+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:42.680 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:42.694+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:42.719 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:42.720 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:42.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:42.783 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:42.796+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:42.862 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:42.862 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:42.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:42.883 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:42.897+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:42.985 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:42.999+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:43.003 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:43.003 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:43.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:43.068 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:21.045 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:43.086 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:43.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:43.114 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:15.427 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:43.114 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:15.429 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:43.128 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:21.107 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:43.137 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:15.449 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:43.160 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:21.139 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:43.187 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:43.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:43.197 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:43.198 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:43.208 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:43.292 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:43.306+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:43.309 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 7, 0, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:27:43.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-06T09:27:43.389 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:43.403+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:43.397 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:43.397 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:43.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:43.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:43.503 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:43.517+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:43.561 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:43.561 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:43.570 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:43.612 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:43.626+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:43.692 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:43.706+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:43.719 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:43.743 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:43.743 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:43.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:43.793 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:43.806+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:43.809 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:43.917 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:43.917 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:43.929 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:43.943+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:43.973 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:43.974 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:43.983 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:44.024 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:44.038+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:44.046 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:44.046 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:44.096 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:44.110+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:44.144 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:44.144 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:44.144 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:44.144 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:44.145 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:44.145 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:44.155 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:44.199 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:44.213+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:44.233 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:27:44.300 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:44.300 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:44.300 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:44.301 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:44.301 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:44.309 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:44.321 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:44.335+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:44.334 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.1 to 1 (10000) 2023-12-06T09:27:44.335 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:44.335 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:44.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-06T09:27:44.399 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:44.412+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:44.422 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:44.479 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:44.479 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:44.479 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:44.489 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:44.500 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:44.513+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:44.600 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:44.614+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:44.661 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:44.661 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:44.661 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:44.662 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:44.671 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:44.701 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:44.716+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:44.803 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:44.817+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:44.804 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:44.804 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:44.804 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:44.804 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:44.804 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:44.814 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:44.904 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:44.918+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 346.27702433399998, 2023-12-06T09:27:44.987 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.988 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:27:44.989 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.990 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:44.991 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.582570465, 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:44.992 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:27:44.993 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:27:44.994 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.994 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.994 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:27:44.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:27:45.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:27:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:27:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:27:45.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:45.002 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.13967798900001, 2023-12-06T09:27:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:27:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:45.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:27:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:27:45.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.008 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.13085170800002, 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.011 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:27:45.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:27:45.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.130230036, 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:27:45.014 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:27:45.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.12228355500002, 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:27:45.018 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:27:45.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:27:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:27:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:27:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:27:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.027 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:27:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:45.028 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.12180953699999, 2023-12-06T09:27:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:27:45.034 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:27:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:27:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 345.121355182, 2023-12-06T09:27:45.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.044 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 344.73736293100001, 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:27:45.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:27:45.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 344.35221352600001, 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.049 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:27:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:27:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:27:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.052 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:27:45.054 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.055 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 344.33993821500002, 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 344.33845263699999, 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:27:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:27:45.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:27:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:27:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:27:45.065 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.431915479, 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:27:45.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.42678317600001, 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.070 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:27:45.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:27:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:27:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.078 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.42595714599997, 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:27:45.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:27:45.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:27:45.083 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.42338768500002, 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.085 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.086 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.087 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.422642215, 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.088 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:27:45.089 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:27:45.090 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:27:45.091 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.421836144, 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.092 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:27:45.093 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.094 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.095 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.096 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 336.41073169200001, 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.097 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.098 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.099 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.100 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:45.101 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 187.637399555, 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:45.102 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:27:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.104 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.105 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:45.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:45.107 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:45.121+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:45.208 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:45.222+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:45.280 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:45.280 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:45.280 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:45.280 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:45.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:45.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:27:45.330 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:45.344+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:45.397 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.2 to 1 (10000) 2023-12-06T09:27:45.411 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:45.424+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:45.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-06T09:27:45.462 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:45.462 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:45.462 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:45.462 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:45.462 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:45.471 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:45.512 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:45.526+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:45.613 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:45.627+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2023-12-06T09:27:45.643 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 30.434090328, 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010005948000000001, 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 11 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.644 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221310+0000", 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6829000000000001e-05 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221505+0000", 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019430599999999999 2023-12-06T09:27:45.645 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221716+0000", 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021168399999999999 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.224250+0000", 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025338729999999999 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.229715+0000", 2023-12-06T09:27:45.646 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054648589999999999 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.231300+0000", 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001584397 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:45.647 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:45.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:45.714 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:45.728+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:45.785 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:45.785 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:45.785 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:45.785 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:45.793 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:45.834 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:45.848+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:45.916 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:45.930+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:45.952 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":38,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:27:45.974 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:45.974 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:45.974 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:45.974 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:45.974 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:45.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:46.030 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:46.044+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:46.113 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:46.114 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:46.114 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:46.117 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:46.131+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:46.124 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:46.219 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:46.233+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:46.261 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:46.262 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:46.262 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:46.262 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:46.270 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:46.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:27:46.354 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:46.368+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:46.364 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.3 to 1 (10000) 2023-12-06T09:27:46.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-12-06T09:27:46.427 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:46.441+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:46.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:46.458 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:46.458 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:46.458 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:46.458 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:46.458 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:46.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:46.521 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:46.535+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:46.622 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:46.636+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:46.652 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:46.653 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:46.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:46.653 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:27:46.653 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:46.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:46.723 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:46.737+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:46.795 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:46.795 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:46.795 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:46.796 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:46.805 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:46.849 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:46.863+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:46.925 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:46.939+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:46.959 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:46.959 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:46.988 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:46.989 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:46.989 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:46.989 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:46.989 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:47.002 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:47.043 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:47.057+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:47.065 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:47.127 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:47.141+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:47.201 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:47.201 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:47.207 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:47.207 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:47.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:47.251 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:47.265+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:47.295 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:47.295 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:47.329 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:47.343+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:47.378 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:47.378 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:47.397 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:47.397 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:47.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:47.464 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:47.465 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:47.478+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.6 to 1 (10000) 2023-12-06T09:27:47.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-12-06T09:27:47.529 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:27:47.558 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:47.559 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:47.559 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:47.571+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:47.608 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:47.608 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:47.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:47.659 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:47.673+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:47.667 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:47.731 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:47.745+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:47.816 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:47.816 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:47.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:47.867 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:47.881+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:47.933 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:47.946+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:48.028 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:48.029 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:48.033 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:48.047+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:48.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:48.134 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:48.148+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:48.222 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:48.222 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:48.231 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:48.278 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:48.289+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:48.337 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:48.350+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:48.376 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:48.376 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:48.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:48.438 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:48.452+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:48.517 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:48.517 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:48.527 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:48.539 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:48.553+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:48.640 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:48.653+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:48.700 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:48.701 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:48.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:48.751 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:48.765+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:48.842 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:48.855+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:48.876 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:48.877 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:48.886 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:48.942 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:48.956+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:48.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:27:49.014 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:49.014 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:49.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:49.065 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:49.079+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:49.144 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:49.158+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:49.177 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.7 to 1 (10000) 2023-12-06T09:27:49.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-06T09:27:49.212 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:49.212 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:49.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:49.245 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:49.259+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:49.346 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:49.360+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:49.393 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:49.394 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:49.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:49.446 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:49.460+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:49.547 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:49.561+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:49.550 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T09:27:49.572 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:49.572 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:49.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:49.648 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:49.662+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:49.651 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":42,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:27:49.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:49.749 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:49.763+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:49.756 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:49.756 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:49.766 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:49.850 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:49.864+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:49.935 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:49.935 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:49.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:49.945 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:49.945 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:49.988 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:50.002+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:50.033 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:50.033 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:50.058 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:50.072+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:50.072 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:50.073 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:50.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:50.154 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:50.168+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:50.164 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:50.165 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:50.258 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.604 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:50.259 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:50.259 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:50.263 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.0 to 1 (10000) 2023-12-06T09:27:50.266 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:50.278+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:50.266 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:50.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:50.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-12-06T09:27:50.364 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:50.378+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:50.370 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:50.371 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:50.416 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:50.416 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:50.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:50.457 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:50.471+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:50.470 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:50.470 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:50.506 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:08.852 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:50.554 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:50.575 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:50.575 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:50.575 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:50.576 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:50.581 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:50.594+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:50.585 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:50.638 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:50.661 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:50.675+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:50.720 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:50.721 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:50.721 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:50.721 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:50.721 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:50.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:50.772 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:50.786+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:50.863 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:50.877+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:50.923 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:50.923 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:50.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:50.923 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2023-12-06T09:27:50.923 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:50.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:50.974 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:50.988+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:51.065 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:51.079+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:51.135 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:51.135 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:51.136 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:51.136 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:51.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:51.166 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:51.180+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:51.268 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:51.281+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:51.319 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:51.319 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:51.320 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:51.320 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:51.320 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:51.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:51.345 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is set 2023-12-06T09:27:51.346 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.5 to 1 (10000) 2023-12-06T09:27:51.365 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-06T09:27:51.366 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:27:51.407 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:51.421+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:51.469 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:51.483+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 354.63404232599999, 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0186807939999998, 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.865808+0000", 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.866490+0000", 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068211300000000003 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.871825+0000", 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053347539999999997 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.876874+0000", 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050494659999999999 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.886009+0000", 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091343599999999994 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.910441+0000", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024431877000000001 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:56.917078+0000", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066369439999999997 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.359977+0000", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4428992950000001 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.442614+0000", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082636713000000001 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.534551+0000", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091936927000000002 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.671273+0000", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13672272299999999 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.857812+0000", 2023-12-06T09:27:51.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.186538912 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.869205+0000", 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011392625999999999 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:58.884489+0000", 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015284084 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.549 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 352.21633220899997, 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22827419199999999, 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:51.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283518+0000", 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.283529+0000", 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1263999999999999e-05 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.286952+0000", 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003422586 2023-12-06T09:27:51.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287119+0000", 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167181 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287278+0000", 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015899999999999999 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.288521+0000", 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001242593 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.290121+0000", 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016002639999999999 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.296982+0000", 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068612200000000003 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.455457+0000", 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15847438699999999 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.510045+0000", 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054588276999999998 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511074+0000", 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010287880000000001 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.511792+0000", 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071863199999999997 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 352.21214776400001, 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.228783706, 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.555 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287703+0000", 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.287712+0000", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5629999999999998e-06 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372617+0000", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084905262999999995 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372773+0000", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015510900000000001 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.372918+0000", 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014584999999999999 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.376148+0000", 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032297720000000001 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.377753+0000", 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016049759999999999 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.382700+0000", 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049465439999999998 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456029+0000", 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073329285999999994 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.514857+0000", 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058827622000000003 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.515797+0000", 2023-12-06T09:27:51.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000940429 2023-12-06T09:27:51.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.516486+0000", 2023-12-06T09:27:51.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00068929199999999999 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 352.126616678, 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14746293899999999, 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.565 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 22 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373234+0000", 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373246+0000", 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2486999999999999e-05 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373380+0000", 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133831 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373526+0000", 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014550500000000001 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.373656+0000", 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013005800000000001 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.391941+0000", 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018285132999999999 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.394410+0000", 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024688710000000001 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.397604+0000", 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031944429999999999 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.456141+0000", 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058536902000000002 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.519072+0000", 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062931036999999995 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520035+0000", 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096339699999999995 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.520697+0000", 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066127500000000001 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.812640063, 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208315957, 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 30 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687210+0000", 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687221+0000", 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0784999999999999e-05 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687379+0000", 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015741 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687529+0000", 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150109 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.687673+0000", 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014407100000000001 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.696062+0000", 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00838939 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.698784+0000", 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027220949999999999 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.703047+0000", 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042626180000000001 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832551+0000", 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.129504654 2023-12-06T09:27:51.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.893725+0000", 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061173063999999999 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.894769+0000", 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010444180000000001 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.895526+0000", 2023-12-06T09:27:51.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075734299999999995 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.81112221900003, 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20923414900000001, 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.577 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 31 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688728+0000", 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688738+0000", 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5799999999999998e-06 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.688867+0000", 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012879999999999999 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689019+0000", 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015258399999999999 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689161+0000", 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014162800000000001 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.709502+0000", 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020340885 2023-12-06T09:27:51.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.711586+0000", 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020838470000000002 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.715018+0000", 2023-12-06T09:27:51.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034321180000000001 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832674+0000", 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11765589799999999 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.896267+0000", 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063593296999999993 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897245+0000", 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097855399999999997 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.897962+0000", 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071695799999999996 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.583 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.81045684399999, 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211070329, 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 32 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689394+0000", 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.689403+0000", 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2580000000000001e-06 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693146+0000", 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037433240000000001 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693283+0000", 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013667900000000001 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693422+0000", 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139016 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.721484+0000", 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028062052000000001 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.723815+0000", 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023308309999999998 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.727302+0000", 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034870510000000001 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.832793+0000", 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10549126 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.898723+0000", 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065930100000000005 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.899733+0000", 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010097680000000001 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.900464+0000", 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073099000000000005 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:51.589 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.806196672, 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.209194613, 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693654+0000", 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693663+0000", 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4259999999999992e-06 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693788+0000", 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012505799999999999 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.693920+0000", 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013218099999999999 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694063+0000", 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014219699999999999 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.737638+0000", 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043575265000000002 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.739324+0000", 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001686122 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.742011+0000", 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026874910000000002 2023-12-06T09:27:51.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.834912+0000", 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092900983000000006 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.901181+0000", 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066268097999999998 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902155+0000", 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097395800000000003 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.902848+0000", 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069383400000000005 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:51.595 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.805563463, 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.211059889, 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 34 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694287+0000", 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694296+0000", 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0909999999999999e-06 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694419+0000", 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012263700000000001 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694551+0000", 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013212400000000001 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694687+0000", 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013609399999999999 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.749644+0000", 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054956917000000001 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.751334+0000", 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016906899999999999 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.754771+0000", 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034362329999999999 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835051+0000", 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080279970000000006 2023-12-06T09:27:51.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.903648+0000", 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068596848000000002 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.904618+0000", 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097017999999999996 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.905347+0000", 2023-12-06T09:27:51.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072910499999999997 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.80493636099999, 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21281008300000001, 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.601 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 35 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694914+0000", 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.694923+0000", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0219999999999993e-06 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782480+0000", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087556788999999996 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782666+0000", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000186453 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.782827+0000", 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016042000000000001 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.787920+0000", 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005093462 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.790350+0000", 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002429587 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.793634+0000", 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032838619999999998 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835153+0000", 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041519674999999999 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.906055+0000", 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070901398000000004 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907028+0000", 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00097333499999999998 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.907724+0000", 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00069607999999999996 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.606 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.71667658299998, 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12696844800000001, 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 36 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783174+0000", 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783186+0000", 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2653000000000001e-05 2023-12-06T09:27:51.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783336+0000", 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014911300000000001 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783488+0000", 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015254999999999999 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.783717+0000", 2023-12-06T09:27:51.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022864200000000001 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.799546+0000", 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015829609000000001 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.801204+0000", 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001658014 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.803815+0000", 2023-12-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026108049999999999 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835254+0000", 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031438982999999997 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.908457+0000", 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073202946000000005 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.909422+0000", 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00096520799999999995 2023-12-06T09:27:51.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910142+0000", 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071992499999999995 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:51.612 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 351.71575560299999, 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128478963, 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 37 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:51.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784095+0000", 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784110+0000", 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5245000000000001e-05 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784319+0000", 2023-12-06T09:27:51.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020891599999999999 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784551+0000", 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00023173599999999999 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.784699+0000", 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014781700000000001 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.807806+0000", 2023-12-06T09:27:51.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023107954 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.809486+0000", 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016792090000000001 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.812088+0000", 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026024569999999999 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.835354+0000", 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023266156999999999 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.910848+0000", 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075493641 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.911873+0000", 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001025438 2023-12-06T09:27:51.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:21:59.912574+0000", 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000700393 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:51.618 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 350.96337767199998, 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124919607, 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 72 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536473+0000", 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.536482+0000", 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6520000000000001e-06 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558308+0000", 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021825124000000001 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558459+0000", 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000151557 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.558608+0000", 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149154 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.561607+0000", 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002998419 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.563281+0000", 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016746949999999999 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.565959+0000", 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026776640000000002 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.613435+0000", 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047476123000000002 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.659576+0000", 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046140875999999997 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.660610+0000", 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010336500000000001 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:00.661392+0000", 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000782693 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.09334270099998, 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171338932, 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.624 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 104 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406508+0000", 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406522+0000", 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4402e-05 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406701+0000", 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017899600000000001 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.406882+0000", 2023-12-06T09:27:51.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018054400000000001 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407060+0000", 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017810999999999999 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.408795+0000", 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001735215 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.411307+0000", 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025124119999999999 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.414631+0000", 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033237359999999999 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.521911+0000", 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107279558 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.574600+0000", 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052689503999999998 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.576502+0000", 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019014489999999999 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.577847+0000", 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013450059999999999 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.09216918999999, 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.181505215, 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407681+0000", 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407691+0000", 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7769999999999997e-06 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.407879+0000", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018811899999999999 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415307+0000", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074278729999999998 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415516+0000", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020880700000000001 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.428771+0000", 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013255365 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.431134+0000", 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023629020000000001 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.434644+0000", 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003510385 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.535953+0000", 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101308458 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.587345+0000", 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051392378000000002 2023-12-06T09:27:51.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.588401+0000", 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010561100000000001 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589186+0000", 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00078504099999999999 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.08401721899997, 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17591322300000001, 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 106 2023-12-06T09:27:51.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415833+0000", 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.415846+0000", 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3003000000000001e-05 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425472+0000", 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096254289999999996 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425641+0000", 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000169051 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.425798+0000", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000157541 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.439468+0000", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013669717 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.441143+0000", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016751019999999999 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.443872+0000", 2023-12-06T09:27:51.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00272894 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536154+0000", 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092282343000000003 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.589954+0000", 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053799481000000003 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.590974+0000", 2023-12-06T09:27:51.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010200090000000001 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.591746+0000", 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077260700000000002 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.07373528300002, 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168099528, 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.641 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 107 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426115+0000", 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426128+0000", 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.254e-05 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426269+0000", 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140891 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426414+0000", 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145372 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426547+0000", 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013302 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.462907+0000", 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036359672000000003 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.464609+0000", 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001701925 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.467392+0000", 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027835500000000001 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536341+0000", 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068948946999999997 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.592489+0000", 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056147649000000001 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.593479+0000", 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099073499999999997 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594215+0000", 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00073522699999999997 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.646 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.07307740300001, 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16994007, 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.647 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426773+0000", 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426782+0000", 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7399999999999993e-06 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.426917+0000", 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013530700000000001 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427071+0000", 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000154186 2023-12-06T09:27:51.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427207+0000", 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013625399999999999 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.471842+0000", 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044634412999999998 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.473489+0000", 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001646914 2023-12-06T09:27:51.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.476387+0000", 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028978559999999999 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.536528+0000", 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060141685 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.594952+0000", 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058423401999999999 2023-12-06T09:27:51.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.595945+0000", 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00099370600000000002 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.596713+0000", 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00076760700000000001 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.652 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.07240262400001, 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17233294800000001, 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 109 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:51.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427448+0000", 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.427457+0000", 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7460000000000006e-06 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.457392+0000", 2023-12-06T09:27:51.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029935801000000001 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459288+0000", 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001895898 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459456+0000", 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016780300000000001 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.480899+0000", 2023-12-06T09:27:51.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021443203000000001 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.482587+0000", 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016877109999999999 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.485362+0000", 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027750380000000001 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537645+0000", 2023-12-06T09:27:51.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052282755 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.597692+0000", 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.060046995999999998 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.598956+0000", 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001263896 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.599781+0000", 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00082510099999999996 2023-12-06T09:27:51.657 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4139.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.04004503300001, 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14252256699999999, 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4139", 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/651229093", 2023-12-06T09:27:51.658 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 110 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459805+0000", 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:51.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.459822+0000", 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6562000000000002e-05 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460067+0000", 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000245472 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460290+0000", 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000222196 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.460456+0000", 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016598000000000001 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.507425+0000", 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046969344000000003 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.510350+0000", 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029246649999999999 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.515066+0000", 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004716334 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.537831+0000", 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022765396 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2023-12-06T09:27:51.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.600557+0000", 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.062726176999999994 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.601609+0000", 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001051621 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:08.602328+0000", 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071882000000000001 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.663 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.664 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:51.664 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:51.664 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:51.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:51.666 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:51.679+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:51.766 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:51.780+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:51.854 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:51.855 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:51.855 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:51.855 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:51.864 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:51.867 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:51.881+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:51.968 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:51.982+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:52.034 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:52.034 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:52.034 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:52.035 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:52.035 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:52.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:52.047 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:27:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:27:52.060 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2023-12-06T09:27:51.979867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":195024,"kb_used_data":4780,"kb_used_omap":13,"kb_used_meta":190002,"kb_avail":660407856,"statfs":{"total":676457349120,"available":676257644544,"internally_reserved":0,"allocated":4894720,"data_stored":2527511,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":13962,"internal_metadata":194562422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.488636"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-12-06T09:27:46.444952+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:27:46.444952+0000","last_peered":"2023-12-06T09:27:46.444952+0000","last_clean":"2023-12-06T09:27:46.444952+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:27:46.444952+0000","last_undegraded":"2023-12-06T09:27:46.444952+0000","last_fullsized":"2023-12-06T09:27:46.444952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-12-06T09:27:46.394679+0000","last_change":"2023-12-06T09:27:30.512697+0000","last_active":"2023-12-06T09:27:46.394679+0000","last_peered":"2023-12-06T09:27:46.394679+0000","last_clean":"2023-12-06T09:27:46.394679+0000","last_became_active":"2023-12-06T09:27:30.506758+0000","last_became_peered":"2023-12-06T09:27:30.506758+0000","last_unstale":"2023-12-06T09:27:46.394679+0000","last_undegraded":"2023-12-06T09:27:46.394679+0000","last_fullsized":"2023-12-06T09:27:46.394679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:05:17.828409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-06T09:27:49.269294+0000","last_change":"2023-12-06T09:27:30.879225+0000","last_active":"2023-12-06T09:27:49.269294+0000","last_peered":"2023-12-06T09:27:49.269294+0000","last_clean":"2023-12-06T09:27:49.269294+0000","last_became_active":"2023-12-06T09:27:30.873153+0000","last_became_peered":"2023-12-06T09:27:30.873153+0000","last_unstale":"2023-12-06T09:27:49.269294+0000","last_undegraded":"2023-12-06T09:27:49.269294+0000","last_fullsized":"2023-12-06T09:27:49.269294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:39:41.323105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-12-06T09:27:45.146940+0000","last_change":"2023-12-06T09:27:23.414170+0000","last_active":"2023-12-06T09:27:45.146940+0000","last_peered":"2023-12-06T09:27:45.146940+0000","last_clean":"2023-12-06T09:27:45.146940+0000","last_became_active":"2023-12-06T09:27:23.408038+0000","last_became_peered":"2023-12-06T09:27:23.408038+0000","last_unstale":"2023-12-06T09:27:45.146940+0000","last_undegraded":"2023-12-06T09:27:45.146940+0000","last_fullsized":"2023-12-06T09:27:45.146940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:21:51.319696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-06T09:27:47.585797+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:27:47.585797+0000","last_peered":"2023-12-06T09:27:47.585797+0000","last_clean":"2023-12-06T09:27:47.585797+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:27:47.585797+0000","last_undegraded":"2023-12-06T09:27:47.585797+0000","last_fullsized":"2023-12-06T09:27:47.585797+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-12-06T09:27:45.143524+0000","last_change":"2023-12-06T09:23:45.621641+0000","last_active":"2023-12-06T09:27:45.143524+0000","last_peered":"2023-12-06T09:27:45.143524+0000","last_clean":"2023-12-06T09:27:45.143524+0000","last_became_active":"2023-12-06T09:23:45.615923+0000","last_became_peered":"2023-12-06T09:23:45.615923+0000","last_unstale":"2023-12-06T09:27:45.143524+0000","last_undegraded":"2023-12-06T09:27:45.143524+0000","last_fullsized":"2023-12-06T09:27:45.143524+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:06:07.386710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-06T09:27:49.260233+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:27:49.260233+0000","last_peered":"2023-12-06T09:27:49.260233+0000","last_clean":"2023-12-06T09:27:49.260233+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:27:49.260233+0000","last_undegraded":"2023-12-06T09:27:49.260233+0000","last_fullsized":"2023-12-06T09:27:49.260233+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-12-06T09:27:45.141785+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:27:45.141785+0000","last_peered":"2023-12-06T09:27:45.141785+0000","last_clean":"2023-12-06T09:27:45.141785+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:27:45.141785+0000","last_undegraded":"2023-12-06T09:27:45.141785+0000","last_fullsized":"2023-12-06T09:27:45.141785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":218,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-06T09:27:47.550335+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:27:47.550335+0000","last_peered":"2023-12-06T09:27:47.550335+0000","last_clean":"2023-12-06T09:27:47.550335+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:27:47.550335+0000","last_undegraded":"2023-12-06T09:27:47.550335+0000","last_fullsized":"2023-12-06T09:27:47.550335+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509514,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":254391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":9.5999999999999996,"5min":10.960000000000001,"15min":14.996},"min":{"1min":4.4269999999999996,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":20.725999999999999,"5min":49.691000000000003,"15min":49.691000000000003},"last":6.2809999999999997},{"interface":"front","average":{"1min":8.8949999999999996,"5min":10.445,"15min":14.975},"min":{"1min":5.8170000000000002,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":16.494,"5min":51.530000000000001,"15min":51.530000000000001},"last":10.731999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.195,"5min":10.938000000000001,"15min":12.022},"min":{"1min":5.7530000000000001,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":40.991,"5min":40.991,"15min":40.991},"last":14.989000000000001},{"interface":"front","average":{"1min":15.298,"5min":13.292,"15min":14.442},"min":{"1min":8.1880000000000006,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":35.850000000000001,"5min":35.850000000000001,"15min":35.850000000000001},"last":9.8209999999999997}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.053000000000001,"5min":12.885,"15min":14.698},"min":{"1min":7.3819999999999997,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":18.181999999999999,"5min":42.523000000000003,"15min":42.523000000000003},"last":12.99},{"interface":"front","average":{"1min":14.087,"5min":15.260999999999999,"15min":17.285},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":21.725999999999999,"5min":39.066000000000003,"15min":39.066000000000003},"last":33.149999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.579000000000001,"5min":13.173,"15min":14.601000000000001},"min":{"1min":9.1319999999999997,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":34.061,"5min":34.061,"15min":34.061},"last":14.175000000000001},{"interface":"front","average":{"1min":19.207999999999998,"5min":15.503,"15min":15.571},"min":{"1min":11.127000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":32.462000000000003,"5min":32.462000000000003,"15min":32.462000000000003},"last":31.831}]},{"osd":5,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.001999999999999,"5min":16.791,"15min":24.204999999999998},"min":{"1min":8.8490000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":24.170999999999999,"5min":190.821,"15min":190.821},"last":18.899999999999999},{"interface":"front","average":{"1min":17.789999999999999,"5min":18.347000000000001,"15min":24.542999999999999},"min":{"1min":6.1950000000000003,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":44.625999999999998,"5min":191.86199999999999,"15min":191.86199999999999},"last":10.471}]},{"osd":6,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":20.004000000000001,"5min":18.93,"15min":24.388999999999999},"min":{"1min":11.297000000000001,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":33.362000000000002,"5min":166.45400000000001,"15min":166.45400000000001},"last":36.244},{"interface":"front","average":{"1min":18.824999999999999,"5min":19.602,"15min":24.550000000000001},"min":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.606000000000002,"5min":117.58499999999999,"15min":117.58499999999999},"last":19.757999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":23.681000000000001,"5min":23.681000000000001,"15min":23.681000000000001},"min":{"1min":12.381,"5min":12.381,"15min":12.381},"max":{"1min":52.109000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":38.101999999999997},{"interface":"front","average":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"min":{"1min":11.01,"5min":11.01,"15min":11.01},"max":{"1min":56.914000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":15.685}]}]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":659456,"data_stored":236607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.0969999999999995,"15min":8.1890000000000001},"min":{"1min":6.0819999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":25.431000000000001,"5min":38.481000000000002,"15min":38.481000000000002},"last":6.1909999999999998},{"interface":"front","average":{"1min":10.859,"5min":11.725,"15min":12.146000000000001},"min":{"1min":5.7930000000000001,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":26.995000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":9.6839999999999993}]},{"osd":1,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":13.384,"15min":14.712},"min":{"1min":9.0299999999999994,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":26.780000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":16.446000000000002},{"interface":"front","average":{"1min":16.044,"5min":16.731999999999999,"15min":18.222000000000001},"min":{"1min":9.6699999999999999,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":28.262,"5min":54.338000000000001,"15min":54.338000000000001},"last":17.684000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":11.717000000000001,"15min":11.206},"min":{"1min":7.7359999999999998,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":21.158000000000001,"5min":27.295999999999999,"15min":27.295999999999999},"last":10.429},{"interface":"front","average":{"1min":11.641,"5min":11.231,"15min":12.781000000000001},"min":{"1min":6.468,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":19.015999999999998,"5min":28.669,"15min":28.669},"last":8.8079999999999998}]},{"osd":4,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.824,"5min":14.007,"15min":14.811},"min":{"1min":7.2889999999999997,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":20.074999999999999,"5min":54.32,"15min":54.32},"last":11.132999999999999},{"interface":"front","average":{"1min":15.898,"5min":16.734000000000002,"15min":17.497},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":26.274000000000001,"5min":51.950000000000003,"15min":51.950000000000003},"last":18.809000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.038,"15min":15.157},"min":{"1min":7.0330000000000004,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":48.563000000000002,"5min":56.491,"15min":56.491},"last":14.550000000000001},{"interface":"front","average":{"1min":16.576000000000001,"5min":16.488,"15min":19.004000000000001},"min":{"1min":9.5960000000000001,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":47.328000000000003,"5min":75.540999999999997,"15min":75.540999999999997},"last":12.085000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.277999999999999,"15min":18.053000000000001},"min":{"1min":10.872999999999999,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":28.962,"5min":30.213999999999999,"15min":30.213999999999999},"last":19.742000000000001},{"interface":"front","average":{"1min":14.414,"5min":13.6,"15min":14.318},"min":{"1min":8.2479999999999993,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":24.893999999999998,"5min":26.312999999999999,"15min":26.312999999999999},"last":16.030999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:58 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":17.641999999999999,"15min":17.641999999999999},"min":{"1min":12.304,"5min":12.304,"15min":12.304},"max":{"1min":30.283000000000001,"5min":30.283000000000001,"15min":30.283000000000001},"last":20.629999999999999},{"interface":"front","average":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":19.783000000000001},"min":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":31.402999999999999,"5min":31.402999999999999,"15min":31.402999999999999},"last":21.498999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":708608,"data_stored":254391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.576000000000001,"5min":11.565,"15min":11.337999999999999},"min":{"1min":4.8289999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":52.207000000000001,"5min":65.808000000000007,"15min":65.808000000000007},"last":8.0340000000000007},{"interface":"front","average":{"1min":12.784000000000001,"5min":12.163,"15min":11.606999999999999},"min":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":53.106999999999999,"5min":53.106999999999999,"15min":53.106999999999999},"last":11.231}]},{"osd":2,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.065,"15min":13.079000000000001},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":41.301000000000002,"5min":49.652999999999999,"15min":49.652999999999999},"last":19.478000000000002},{"interface":"front","average":{"1min":12.539,"5min":13.265000000000001,"15min":14.048},"min":{"1min":7.9960000000000004,"5min":6.798,"15min":6.798},"max":{"1min":21.843,"5min":47.484000000000002,"15min":47.484000000000002},"last":26.891999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.029,"5min":13.706,"15min":20.071999999999999},"min":{"1min":6.7320000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":32.063000000000002,"5min":205.096,"15min":205.096},"last":9.7249999999999996},{"interface":"front","average":{"1min":18.690000000000001,"5min":17.489000000000001,"15min":22.172000000000001},"min":{"1min":8.5660000000000007,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":45.463000000000001,"5min":182.74000000000001,"15min":182.74000000000001},"last":21.399999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.943,"5min":13.228,"15min":13.978},"min":{"1min":8.0660000000000007,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":29.585999999999999,"5min":45.112000000000002,"15min":45.112000000000002},"last":23.762},{"interface":"front","average":{"1min":17.719000000000001,"5min":16.870999999999999,"15min":16.803000000000001},"min":{"1min":7.1470000000000002,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":34.533000000000001,"5min":52.655999999999999,"15min":52.655999999999999},"last":22.329000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":15.237,"5min":15.542,"15min":15.226000000000001},"min":{"1min":9.2129999999999992,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":25.379999999999999,"5min":43.088000000000001,"15min":43.088000000000001},"last":29.289000000000001},{"interface":"front","average":{"1min":16.452999999999999,"5min":16.266999999999999,"15min":16.77},"min":{"1min":9.7230000000000008,"5min":8.266,"15min":8.266},"max":{"1min":31.550000000000001,"5min":45.722000000000001,"15min":45.722000000000001},"last":26.245999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":21.395,"5min":19.311,"15min":20.494},"min":{"1min":9.2309999999999999,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":44.292999999999999,"5min":78.338999999999999,"15min":78.338999999999999},"last":24.547999999999998},{"interface":"front","average":{"1min":18.216000000000001,"5min":17.864999999999998,"15min":20.274000000000001},"min":{"1min":10.634,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":42.417999999999999,"5min":80.850999999999999,"15min":80.850999999999999},"last":34.250999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":18.908999999999999,"5min":18.908999999999999,"15min":18.908999999999999},"min":{"1min":8.2469999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":61.445999999999998,"5min":61.445999999999998,"15min":61.445999999999998},"last":27.923999999999999},{"interface":"front","average":{"1min":20.067,"5min":20.067,"15min":20.067},"min":{"1min":11.833,"5min":11.833,"15min":11.833},"max":{"1min":63.518000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":32.401000000000003}]}]},{"osd":7,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":668,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":684032,"data_stored":245499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":11.984999999999999,"5min":11.984999999999999,"15min":11.984999999999999},"min":{"1min":4.452,"5min":4.452,"15min":4.452},"max":{"1min":55.622,"5min":55.622,"15min":55.622},"last":5.952},{"interface":"front","average":{"1min":12.089,"5min":12.089,"15min":12.089},"min":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":31.178000000000001,"5min":31.178000000000001,"15min":31.178000000000001},"last":13.695}]},{"osd":1,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":48.959000000000003,"5min":48.959000000000003,"15min":48.959000000000003},"last":18.742000000000001},{"interface":"front","average":{"1min":13.994999999999999,"5min":13.994999999999999,"15min":13.994999999999999},"min":{"1min":8.0380000000000003,"5min":8.0380000000000003,"15min":8.0380000000000003},"max":{"1min":38.232999999999997,"5min":38.232999999999997,"15min":38.232999999999997},"last":19.106999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":12.195,"5min":12.195,"15min":12.195},"min":{"1min":5.952,"5min":5.952,"15min":5.952},"max":{"1min":20.297999999999998,"5min":20.297999999999998,"15min":20.297999999999998},"last":16.515000000000001},{"interface":"front","average":{"1min":15.340999999999999,"5min":15.340999999999999,"15min":15.340999999999999},"min":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":48.240000000000002,"5min":48.240000000000002,"15min":48.240000000000002},"last":21.501999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":13.568,"15min":13.568},"min":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"max":{"1min":54.603999999999999,"5min":54.603999999999999,"15min":54.603999999999999},"last":15.228},{"interface":"front","average":{"1min":21.209,"5min":21.209,"15min":21.209},"min":{"1min":12.032999999999999,"5min":12.032999999999999,"15min":12.032999999999999},"max":{"1min":52.719000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":20.754999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.044,"5min":15.044,"15min":15.044},"min":{"1min":7.2329999999999997,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":32.390999999999998,"5min":32.390999999999998,"15min":32.390999999999998},"last":22.202999999999999},{"interface":"front","average":{"1min":13.667999999999999,"5min":13.667999999999999,"15min":13.667999999999999},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":15.859}]},{"osd":5,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.632999999999999,"5min":19.632999999999999,"15min":19.632999999999999},"min":{"1min":12.234,"5min":12.234,"15min":12.234},"max":{"1min":52.197000000000003,"5min":52.197000000000003,"15min":52.197000000000003},"last":22.779},{"interface":"front","average":{"1min":17.359999999999999,"5min":17.359999999999999,"15min":17.359999999999999},"min":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":38.877000000000002,"5min":38.877000000000002,"15min":38.877000000000002},"last":27.617000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.231000000000002,"5min":19.231000000000002,"15min":19.231000000000002},"min":{"1min":12.494999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":39.508000000000003,"5min":39.508000000000003,"15min":39.508000000000003},"last":24.303000000000001},{"interface":"front","average":{"1min":14.912000000000001,"5min":14.912000000000001,"15min":14.912000000000001},"min":{"1min":8.6530000000000005,"5min":8.6530000000000005,"15min":8.6530000000000005},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":19.785}]}]},{"osd":6,"up_from":14,"seq":60129542217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":245499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":8.9559999999999995,"5min":10.226000000000001,"15min":11.521000000000001},"min":{"1min":4.3490000000000002,"5min":3.484,"15min":3.484},"max":{"1min":16.178999999999998,"5min":45.506,"15min":45.506},"last":14.35},{"interface":"front","average":{"1min":9.0839999999999996,"5min":10.619999999999999,"15min":12.609999999999999},"min":{"1min":5.1120000000000001,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":16.661000000000001,"5min":46.450000000000003,"15min":46.450000000000003},"last":13.191000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":11.772,"5min":13.038,"15min":15.023999999999999},"min":{"1min":7.4740000000000002,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":20.497,"5min":55.398000000000003,"15min":55.398000000000003},"last":14.965},{"interface":"front","average":{"1min":13.172000000000001,"5min":13.750999999999999,"15min":14.923},"min":{"1min":8.9809999999999999,"5min":6.569,"15min":6.569},"max":{"1min":22.056999999999999,"5min":41.811999999999998,"15min":41.811999999999998},"last":11.449999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":15.488,"15min":20.422000000000001},"min":{"1min":11.131,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":23.66,"5min":208.93799999999999,"15min":208.93799999999999},"last":9.2370000000000001},{"interface":"front","average":{"1min":13.587,"5min":16.247,"15min":24.045999999999999},"min":{"1min":7.3310000000000004,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":24.765000000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":8.3369999999999997}]},{"osd":3,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":17.373999999999999,"5min":15.544,"15min":17.196999999999999},"min":{"1min":10.898999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":25.806999999999999,"5min":43.276000000000003,"15min":43.276000000000003},"last":17.100999999999999},{"interface":"front","average":{"1min":9.9079999999999995,"5min":10.916,"15min":14.124000000000001},"min":{"1min":6.4320000000000004,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":17.137,"5min":41.493000000000002,"15min":41.493000000000002},"last":6.9080000000000004}]},{"osd":4,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":16.132000000000001,"5min":16.742999999999999,"15min":22.754999999999999},"min":{"1min":9.4000000000000004,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":27.443999999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":18.021000000000001},{"interface":"front","average":{"1min":18.768000000000001,"5min":18.789000000000001,"15min":25.478000000000002},"min":{"1min":8.4570000000000007,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":29.385000000000002,"5min":111.874,"15min":111.874},"last":16.210000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":12.522,"5min":12.273,"15min":16.116},"min":{"1min":7.3929999999999998,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":22.010000000000002,"5min":39.886000000000003,"15min":39.886000000000003},"last":10.478999999999999},{"interface":"front","average":{"1min":17.369,"5min":17.016999999999999,"15min":21.588999999999999},"min":{"1min":11.013999999999999,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":30.199999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":20.442}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":20.678999999999998,"5min":20.678999999999998,"15min":20.678999999999998},"min":{"1min":13.064,"5min":13.064,"15min":13.064},"max":{"1min":31.757999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":18.939},{"interface":"front","average":{"1min":15.526999999999999,"5min":15.526999999999999,"15min":15.526999999999999},"min":{"1min":9.6890000000000001,"5min":9.6890000000000001,"15min":9.6890000000000001},"max":{"1min":28.809000000000001,"5min":28.809000000000001,"15min":28.809000000000001},"last":12.446}]}]},{"osd":5,"up_from":14,"seq":60129542217,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":643339,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":9.0600000000000005,"5min":10.461,"15min":13.067},"min":{"1min":4.5670000000000002,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":18.443000000000001,"5min":45.491,"15min":45.491},"last":4.7530000000000001},{"interface":"front","average":{"1min":8.7940000000000005,"5min":10.765000000000001,"15min":12.897},"min":{"1min":4.4349999999999996,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":14.475,"5min":49.991999999999997,"15min":49.991999999999997},"last":7.5289999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":11.146000000000001,"5min":13.614000000000001,"15min":21.236000000000001},"min":{"1min":5.6900000000000004,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":18.754000000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":13.528},{"interface":"front","average":{"1min":10.789,"5min":17.363,"15min":29.614999999999998},"min":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":17.491,"5min":344.79599999999999,"15min":344.79599999999999},"last":9.3339999999999996}]},{"osd":2,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.129,"5min":12.884,"15min":13.379},"min":{"1min":9.1910000000000007,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":24.969999999999999,"5min":53.235999999999997,"15min":53.235999999999997},"last":12.98},{"interface":"front","average":{"1min":14.458,"5min":13.545999999999999,"15min":15.108000000000001},"min":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":26.155000000000001,"5min":54.747,"15min":54.747},"last":8.4130000000000003}]},{"osd":3,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.762,"5min":14.917,"15min":13.975},"min":{"1min":9.6069999999999993,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":24.18,"5min":55.310000000000002,"15min":55.310000000000002},"last":14.596},{"interface":"front","average":{"1min":11.239000000000001,"5min":12.801,"15min":16.099},"min":{"1min":6.9889999999999999,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":19.765000000000001,"5min":51.183,"15min":51.183},"last":10.991}]},{"osd":4,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":12.808,"5min":14.959,"15min":19.792999999999999},"min":{"1min":9.0570000000000004,"5min":6.085,"15min":6.085},"max":{"1min":21.306999999999999,"5min":281.608,"15min":281.608},"last":12.676},{"interface":"front","average":{"1min":16.542999999999999,"5min":16.850999999999999,"15min":21.760999999999999},"min":{"1min":11.683999999999999,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":23.295999999999999,"5min":225.83600000000001,"15min":225.83600000000001},"last":17.292000000000002}]},{"osd":6,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":16.638999999999999,"5min":16.809000000000001,"15min":20.645},"min":{"1min":11.105,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":22.853000000000002,"5min":198.667,"15min":198.667},"last":18.283999999999999},{"interface":"front","average":{"1min":16.501000000000001,"5min":17.471,"15min":22.067},"min":{"1min":10.128,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":24.225000000000001,"5min":208.446,"15min":208.446},"last":15.842000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":14.349,"5min":14.349,"15min":14.349},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":9.9410000000000007},{"interface":"front","average":{"1min":18.501000000000001,"5min":18.501000000000001,"15min":18.501000000000001},"min":{"1min":12.881,"5min":12.881,"15min":12.881},"max":{"1min":25.683,"5min":25.683,"15min":25.683},"last":15.212999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542217,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":733184,"data_stored":647785,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.8759999999999994,"5min":8.9649999999999999,"15min":10.57},"min":{"1min":5.6459999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":26.015000000000001,"5min":40.225999999999999,"15min":40.225999999999999},"last":11.645},{"interface":"front","average":{"1min":11.209,"5min":10.295999999999999,"15min":11.488},"min":{"1min":5.8639999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":20.613,"5min":30.405000000000001,"15min":30.405000000000001},"last":10.731}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":10.425000000000001,"5min":13.679,"15min":11.686},"min":{"1min":7.0549999999999997,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":21.376000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":9.5649999999999995},{"interface":"front","average":{"1min":15.489000000000001,"5min":17.225999999999999,"15min":16.045999999999999},"min":{"1min":10.004,"5min":6.415,"15min":6.415},"max":{"1min":30.391999999999999,"5min":164.911,"15min":164.911},"last":16.396999999999998}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.678000000000001,"5min":14.433999999999999,"15min":19.654},"min":{"1min":5.359,"5min":5.359,"15min":5.359},"max":{"1min":20.265999999999998,"5min":122.762,"15min":122.762},"last":28.663},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.653,"15min":19.021000000000001},"min":{"1min":7.1059999999999999,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":22.109000000000002,"5min":129.99100000000001,"15min":129.99100000000001},"last":25.053999999999998}]},{"osd":4,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.468999999999999,"5min":18.315000000000001,"15min":27.951000000000001},"min":{"1min":9.1140000000000008,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":49.097000000000001,"5min":359.46300000000002,"15min":359.46300000000002},"last":9.1859999999999999},{"interface":"front","average":{"1min":15.288,"5min":17.658999999999999,"15min":26.722999999999999},"min":{"1min":6.4539999999999997,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":47.192,"5min":314.47699999999998,"15min":314.47699999999998},"last":6.7389999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":19.167999999999999,"5min":21.059999999999999,"15min":25.846},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":81.923000000000002,"5min":152.97399999999999,"15min":152.97399999999999},"last":13.445},{"interface":"front","average":{"1min":16.448,"5min":18.698,"15min":24.085000000000001},"min":{"1min":10.108000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.151000000000003,"5min":154.53100000000001,"15min":154.53100000000001},"last":12.574999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.266999999999999,"5min":15.220000000000001,"15min":17.032},"min":{"1min":8.218,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":20.634,"5min":43.045000000000002,"15min":43.045000000000002},"last":17.545000000000002},{"interface":"front","average":{"1min":16.388000000000002,"5min":17.977,"15min":20.286999999999999},"min":{"1min":11.009,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":27.966999999999999,"5min":41.356000000000002,"15min":41.356000000000002},"last":18.413}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":19.687999999999999,"15min":19.687999999999999},"min":{"1min":11.547000000000001,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":32.191000000000003,"5min":32.191000000000003,"15min":32.191000000000003},"last":15.148},{"interface":"front","average":{"1min":19.468,"5min":19.468,"15min":19.468},"min":{"1min":13.093,"5min":13.093,"15min":13.093},"max":{"1min":36.454000000000001,"5min":36.454000000000001,"15min":36.454000000000001},"last":14.021000000000001}]}]},{"osd":4,"up_from":0,"seq":55834574923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:27:52.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:27:52.068 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:52.082+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:52.170 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:52.184+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:52.242 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.60433778199999, 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:52.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:52.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:52.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.245 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 201.299116481, 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:52.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 36.415378689000001, 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029206599999999998, 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 12 2023-12-06T09:27:52.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835047+0000", 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5968999999999999e-05 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:52.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835217+0000", 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017016899999999999 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835416+0000", 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000198677 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835585+0000", 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016903299999999999 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.836489+0000", 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00090463399999999997 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.837952+0000", 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001462178 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.255 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.256 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:52.256 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:52.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:52.301 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:52.315+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:52.372 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:52.386+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:52.390 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:52.390 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:52.391 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:52.391 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:52.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:52.482 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:52.496+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:52.574 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:52.588+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:52.587 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:52.588 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:52.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:52.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:52.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:27:52.675 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:52.689+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:52.681 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:27:52.681 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:27:52.696 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2023-12-06T09:27:51.979867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":195024,"kb_used_data":4780,"kb_used_omap":13,"kb_used_meta":190002,"kb_avail":660407856,"statfs":{"total":676457349120,"available":676257644544,"internally_reserved":0,"allocated":4894720,"data_stored":2527511,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":13962,"internal_metadata":194562422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.488636"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-12-06T09:27:46.444952+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:27:46.444952+0000","last_peered":"2023-12-06T09:27:46.444952+0000","last_clean":"2023-12-06T09:27:46.444952+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:27:46.444952+0000","last_undegraded":"2023-12-06T09:27:46.444952+0000","last_fullsized":"2023-12-06T09:27:46.444952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-12-06T09:27:46.394679+0000","last_change":"2023-12-06T09:27:30.512697+0000","last_active":"2023-12-06T09:27:46.394679+0000","last_peered":"2023-12-06T09:27:46.394679+0000","last_clean":"2023-12-06T09:27:46.394679+0000","last_became_active":"2023-12-06T09:27:30.506758+0000","last_became_peered":"2023-12-06T09:27:30.506758+0000","last_unstale":"2023-12-06T09:27:46.394679+0000","last_undegraded":"2023-12-06T09:27:46.394679+0000","last_fullsized":"2023-12-06T09:27:46.394679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:05:17.828409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-06T09:27:49.269294+0000","last_change":"2023-12-06T09:27:30.879225+0000","last_active":"2023-12-06T09:27:49.269294+0000","last_peered":"2023-12-06T09:27:49.269294+0000","last_clean":"2023-12-06T09:27:49.269294+0000","last_became_active":"2023-12-06T09:27:30.873153+0000","last_became_peered":"2023-12-06T09:27:30.873153+0000","last_unstale":"2023-12-06T09:27:49.269294+0000","last_undegraded":"2023-12-06T09:27:49.269294+0000","last_fullsized":"2023-12-06T09:27:49.269294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:39:41.323105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-12-06T09:27:45.146940+0000","last_change":"2023-12-06T09:27:23.414170+0000","last_active":"2023-12-06T09:27:45.146940+0000","last_peered":"2023-12-06T09:27:45.146940+0000","last_clean":"2023-12-06T09:27:45.146940+0000","last_became_active":"2023-12-06T09:27:23.408038+0000","last_became_peered":"2023-12-06T09:27:23.408038+0000","last_unstale":"2023-12-06T09:27:45.146940+0000","last_undegraded":"2023-12-06T09:27:45.146940+0000","last_fullsized":"2023-12-06T09:27:45.146940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:21:51.319696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-06T09:27:47.585797+0000","last_change":"2023-12-06T09:23:09.133602+0000","last_active":"2023-12-06T09:27:47.585797+0000","last_peered":"2023-12-06T09:27:47.585797+0000","last_clean":"2023-12-06T09:27:47.585797+0000","last_became_active":"2023-12-06T09:23:09.103423+0000","last_became_peered":"2023-12-06T09:23:09.103423+0000","last_unstale":"2023-12-06T09:27:47.585797+0000","last_undegraded":"2023-12-06T09:27:47.585797+0000","last_fullsized":"2023-12-06T09:27:47.585797+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:01:11.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-12-06T09:27:45.143524+0000","last_change":"2023-12-06T09:23:45.621641+0000","last_active":"2023-12-06T09:27:45.143524+0000","last_peered":"2023-12-06T09:27:45.143524+0000","last_clean":"2023-12-06T09:27:45.143524+0000","last_became_active":"2023-12-06T09:23:45.615923+0000","last_became_peered":"2023-12-06T09:23:45.615923+0000","last_unstale":"2023-12-06T09:27:45.143524+0000","last_undegraded":"2023-12-06T09:27:45.143524+0000","last_fullsized":"2023-12-06T09:27:45.143524+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:06:07.386710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-06T09:27:49.260233+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:27:49.260233+0000","last_peered":"2023-12-06T09:27:49.260233+0000","last_clean":"2023-12-06T09:27:49.260233+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:27:49.260233+0000","last_undegraded":"2023-12-06T09:27:49.260233+0000","last_fullsized":"2023-12-06T09:27:49.260233+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-12-06T09:27:45.141785+0000","last_change":"2023-12-06T09:23:09.085600+0000","last_active":"2023-12-06T09:27:45.141785+0000","last_peered":"2023-12-06T09:27:45.141785+0000","last_clean":"2023-12-06T09:27:45.141785+0000","last_became_active":"2023-12-06T09:23:09.045154+0000","last_became_peered":"2023-12-06T09:23:09.045154+0000","last_unstale":"2023-12-06T09:27:45.141785+0000","last_undegraded":"2023-12-06T09:27:45.141785+0000","last_fullsized":"2023-12-06T09:27:45.141785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:44:17.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":218,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-06T09:27:47.550335+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:27:47.550335+0000","last_peered":"2023-12-06T09:27:47.550335+0000","last_clean":"2023-12-06T09:27:47.550335+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:27:47.550335+0000","last_undegraded":"2023-12-06T09:27:47.550335+0000","last_fullsized":"2023-12-06T09:27:47.550335+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509514,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":254391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":9.5999999999999996,"5min":10.960000000000001,"15min":14.996},"min":{"1min":4.4269999999999996,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":20.725999999999999,"5min":49.691000000000003,"15min":49.691000000000003},"last":6.2809999999999997},{"interface":"front","average":{"1min":8.8949999999999996,"5min":10.445,"15min":14.975},"min":{"1min":5.8170000000000002,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":16.494,"5min":51.530000000000001,"15min":51.530000000000001},"last":10.731999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.195,"5min":10.938000000000001,"15min":12.022},"min":{"1min":5.7530000000000001,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":40.991,"5min":40.991,"15min":40.991},"last":14.989000000000001},{"interface":"front","average":{"1min":15.298,"5min":13.292,"15min":14.442},"min":{"1min":8.1880000000000006,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":35.850000000000001,"5min":35.850000000000001,"15min":35.850000000000001},"last":9.8209999999999997}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":11.053000000000001,"5min":12.885,"15min":14.698},"min":{"1min":7.3819999999999997,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":18.181999999999999,"5min":42.523000000000003,"15min":42.523000000000003},"last":12.99},{"interface":"front","average":{"1min":14.087,"5min":15.260999999999999,"15min":17.285},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":21.725999999999999,"5min":39.066000000000003,"15min":39.066000000000003},"last":33.149999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.579000000000001,"5min":13.173,"15min":14.601000000000001},"min":{"1min":9.1319999999999997,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":34.061,"5min":34.061,"15min":34.061},"last":14.175000000000001},{"interface":"front","average":{"1min":19.207999999999998,"5min":15.503,"15min":15.571},"min":{"1min":11.127000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"max":{"1min":32.462000000000003,"5min":32.462000000000003,"15min":32.462000000000003},"last":31.831}]},{"osd":5,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.001999999999999,"5min":16.791,"15min":24.204999999999998},"min":{"1min":8.8490000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":24.170999999999999,"5min":190.821,"15min":190.821},"last":18.899999999999999},{"interface":"front","average":{"1min":17.789999999999999,"5min":18.347000000000001,"15min":24.542999999999999},"min":{"1min":6.1950000000000003,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":44.625999999999998,"5min":191.86199999999999,"15min":191.86199999999999},"last":10.471}]},{"osd":6,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":20.004000000000001,"5min":18.93,"15min":24.388999999999999},"min":{"1min":11.297000000000001,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":33.362000000000002,"5min":166.45400000000001,"15min":166.45400000000001},"last":36.244},{"interface":"front","average":{"1min":18.824999999999999,"5min":19.602,"15min":24.550000000000001},"min":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.606000000000002,"5min":117.58499999999999,"15min":117.58499999999999},"last":19.757999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":23.681000000000001,"5min":23.681000000000001,"15min":23.681000000000001},"min":{"1min":12.381,"5min":12.381,"15min":12.381},"max":{"1min":52.109000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":38.101999999999997},{"interface":"front","average":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"min":{"1min":11.01,"5min":11.01,"15min":11.01},"max":{"1min":56.914000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":15.685}]}]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":659456,"data_stored":236607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.0969999999999995,"15min":8.1890000000000001},"min":{"1min":6.0819999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":25.431000000000001,"5min":38.481000000000002,"15min":38.481000000000002},"last":6.1909999999999998},{"interface":"front","average":{"1min":10.859,"5min":11.725,"15min":12.146000000000001},"min":{"1min":5.7930000000000001,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":26.995000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":9.6839999999999993}]},{"osd":1,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":13.384,"15min":14.712},"min":{"1min":9.0299999999999994,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":26.780000000000001,"5min":52.530000000000001,"15min":52.530000000000001},"last":16.446000000000002},{"interface":"front","average":{"1min":16.044,"5min":16.731999999999999,"15min":18.222000000000001},"min":{"1min":9.6699999999999999,"5min":9.2240000000000002,"15min":9.2240000000000002},"max":{"1min":28.262,"5min":54.338000000000001,"15min":54.338000000000001},"last":17.684000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":11.717000000000001,"15min":11.206},"min":{"1min":7.7359999999999998,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":21.158000000000001,"5min":27.295999999999999,"15min":27.295999999999999},"last":10.429},{"interface":"front","average":{"1min":11.641,"5min":11.231,"15min":12.781000000000001},"min":{"1min":6.468,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":19.015999999999998,"5min":28.669,"15min":28.669},"last":8.8079999999999998}]},{"osd":4,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.824,"5min":14.007,"15min":14.811},"min":{"1min":7.2889999999999997,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":20.074999999999999,"5min":54.32,"15min":54.32},"last":11.132999999999999},{"interface":"front","average":{"1min":15.898,"5min":16.734000000000002,"15min":17.497},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":26.274000000000001,"5min":51.950000000000003,"15min":51.950000000000003},"last":18.809000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.038,"15min":15.157},"min":{"1min":7.0330000000000004,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":48.563000000000002,"5min":56.491,"15min":56.491},"last":14.550000000000001},{"interface":"front","average":{"1min":16.576000000000001,"5min":16.488,"15min":19.004000000000001},"min":{"1min":9.5960000000000001,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":47.328000000000003,"5min":75.540999999999997,"15min":75.540999999999997},"last":12.085000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.277999999999999,"15min":18.053000000000001},"min":{"1min":10.872999999999999,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":28.962,"5min":30.213999999999999,"15min":30.213999999999999},"last":19.742000000000001},{"interface":"front","average":{"1min":14.414,"5min":13.6,"15min":14.318},"min":{"1min":8.2479999999999993,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":24.893999999999998,"5min":26.312999999999999,"15min":26.312999999999999},"last":16.030999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:26:58 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":17.641999999999999,"15min":17.641999999999999},"min":{"1min":12.304,"5min":12.304,"15min":12.304},"max":{"1min":30.283000000000001,"5min":30.283000000000001,"15min":30.283000000000001},"last":20.629999999999999},{"interface":"front","average":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":19.783000000000001},"min":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":31.402999999999999,"5min":31.402999999999999,"15min":31.402999999999999},"last":21.498999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":708608,"data_stored":254391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.576000000000001,"5min":11.565,"15min":11.337999999999999},"min":{"1min":4.8289999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":52.207000000000001,"5min":65.808000000000007,"15min":65.808000000000007},"last":8.0340000000000007},{"interface":"front","average":{"1min":12.784000000000001,"5min":12.163,"15min":11.606999999999999},"min":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":53.106999999999999,"5min":53.106999999999999,"15min":53.106999999999999},"last":11.231}]},{"osd":2,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":13.895,"5min":13.065,"15min":13.079000000000001},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":41.301000000000002,"5min":49.652999999999999,"15min":49.652999999999999},"last":19.478000000000002},{"interface":"front","average":{"1min":12.539,"5min":13.265000000000001,"15min":14.048},"min":{"1min":7.9960000000000004,"5min":6.798,"15min":6.798},"max":{"1min":21.843,"5min":47.484000000000002,"15min":47.484000000000002},"last":26.891999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.029,"5min":13.706,"15min":20.071999999999999},"min":{"1min":6.7320000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":32.063000000000002,"5min":205.096,"15min":205.096},"last":9.7249999999999996},{"interface":"front","average":{"1min":18.690000000000001,"5min":17.489000000000001,"15min":22.172000000000001},"min":{"1min":8.5660000000000007,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":45.463000000000001,"5min":182.74000000000001,"15min":182.74000000000001},"last":21.399999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.943,"5min":13.228,"15min":13.978},"min":{"1min":8.0660000000000007,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":29.585999999999999,"5min":45.112000000000002,"15min":45.112000000000002},"last":23.762},{"interface":"front","average":{"1min":17.719000000000001,"5min":16.870999999999999,"15min":16.803000000000001},"min":{"1min":7.1470000000000002,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":34.533000000000001,"5min":52.655999999999999,"15min":52.655999999999999},"last":22.329000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":15.237,"5min":15.542,"15min":15.226000000000001},"min":{"1min":9.2129999999999992,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":25.379999999999999,"5min":43.088000000000001,"15min":43.088000000000001},"last":29.289000000000001},{"interface":"front","average":{"1min":16.452999999999999,"5min":16.266999999999999,"15min":16.77},"min":{"1min":9.7230000000000008,"5min":8.266,"15min":8.266},"max":{"1min":31.550000000000001,"5min":45.722000000000001,"15min":45.722000000000001},"last":26.245999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:04 2023","interfaces":[{"interface":"back","average":{"1min":21.395,"5min":19.311,"15min":20.494},"min":{"1min":9.2309999999999999,"5min":7.5640000000000001,"15min":7.5640000000000001},"max":{"1min":44.292999999999999,"5min":78.338999999999999,"15min":78.338999999999999},"last":24.547999999999998},{"interface":"front","average":{"1min":18.216000000000001,"5min":17.864999999999998,"15min":20.274000000000001},"min":{"1min":10.634,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":42.417999999999999,"5min":80.850999999999999,"15min":80.850999999999999},"last":34.250999999999998}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":18.908999999999999,"5min":18.908999999999999,"15min":18.908999999999999},"min":{"1min":8.2469999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":61.445999999999998,"5min":61.445999999999998,"15min":61.445999999999998},"last":27.923999999999999},{"interface":"front","average":{"1min":20.067,"5min":20.067,"15min":20.067},"min":{"1min":11.833,"5min":11.833,"15min":11.833},"max":{"1min":63.518000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":32.401000000000003}]}]},{"osd":7,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":668,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":684032,"data_stored":245499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":11.984999999999999,"5min":11.984999999999999,"15min":11.984999999999999},"min":{"1min":4.452,"5min":4.452,"15min":4.452},"max":{"1min":55.622,"5min":55.622,"15min":55.622},"last":5.952},{"interface":"front","average":{"1min":12.089,"5min":12.089,"15min":12.089},"min":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":31.178000000000001,"5min":31.178000000000001,"15min":31.178000000000001},"last":13.695}]},{"osd":1,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":48.959000000000003,"5min":48.959000000000003,"15min":48.959000000000003},"last":18.742000000000001},{"interface":"front","average":{"1min":13.994999999999999,"5min":13.994999999999999,"15min":13.994999999999999},"min":{"1min":8.0380000000000003,"5min":8.0380000000000003,"15min":8.0380000000000003},"max":{"1min":38.232999999999997,"5min":38.232999999999997,"15min":38.232999999999997},"last":19.106999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":12.195,"5min":12.195,"15min":12.195},"min":{"1min":5.952,"5min":5.952,"15min":5.952},"max":{"1min":20.297999999999998,"5min":20.297999999999998,"15min":20.297999999999998},"last":16.515000000000001},{"interface":"front","average":{"1min":15.340999999999999,"5min":15.340999999999999,"15min":15.340999999999999},"min":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":48.240000000000002,"5min":48.240000000000002,"15min":48.240000000000002},"last":21.501999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":13.568,"15min":13.568},"min":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"max":{"1min":54.603999999999999,"5min":54.603999999999999,"15min":54.603999999999999},"last":15.228},{"interface":"front","average":{"1min":21.209,"5min":21.209,"15min":21.209},"min":{"1min":12.032999999999999,"5min":12.032999999999999,"15min":12.032999999999999},"max":{"1min":52.719000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":20.754999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":15.044,"5min":15.044,"15min":15.044},"min":{"1min":7.2329999999999997,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":32.390999999999998,"5min":32.390999999999998,"15min":32.390999999999998},"last":22.202999999999999},{"interface":"front","average":{"1min":13.667999999999999,"5min":13.667999999999999,"15min":13.667999999999999},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":15.859}]},{"osd":5,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.632999999999999,"5min":19.632999999999999,"15min":19.632999999999999},"min":{"1min":12.234,"5min":12.234,"15min":12.234},"max":{"1min":52.197000000000003,"5min":52.197000000000003,"15min":52.197000000000003},"last":22.779},{"interface":"front","average":{"1min":17.359999999999999,"5min":17.359999999999999,"15min":17.359999999999999},"min":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":38.877000000000002,"5min":38.877000000000002,"15min":38.877000000000002},"last":27.617000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.231000000000002,"5min":19.231000000000002,"15min":19.231000000000002},"min":{"1min":12.494999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":39.508000000000003,"5min":39.508000000000003,"15min":39.508000000000003},"last":24.303000000000001},{"interface":"front","average":{"1min":14.912000000000001,"5min":14.912000000000001,"15min":14.912000000000001},"min":{"1min":8.6530000000000005,"5min":8.6530000000000005,"15min":8.6530000000000005},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":19.785}]}]},{"osd":6,"up_from":14,"seq":60129542217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":245499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":8.9559999999999995,"5min":10.226000000000001,"15min":11.521000000000001},"min":{"1min":4.3490000000000002,"5min":3.484,"15min":3.484},"max":{"1min":16.178999999999998,"5min":45.506,"15min":45.506},"last":14.35},{"interface":"front","average":{"1min":9.0839999999999996,"5min":10.619999999999999,"15min":12.609999999999999},"min":{"1min":5.1120000000000001,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":16.661000000000001,"5min":46.450000000000003,"15min":46.450000000000003},"last":13.191000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":11.772,"5min":13.038,"15min":15.023999999999999},"min":{"1min":7.4740000000000002,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":20.497,"5min":55.398000000000003,"15min":55.398000000000003},"last":14.965},{"interface":"front","average":{"1min":13.172000000000001,"5min":13.750999999999999,"15min":14.923},"min":{"1min":8.9809999999999999,"5min":6.569,"15min":6.569},"max":{"1min":22.056999999999999,"5min":41.811999999999998,"15min":41.811999999999998},"last":11.449999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":15.488,"15min":20.422000000000001},"min":{"1min":11.131,"5min":5.1420000000000003,"15min":5.1420000000000003},"max":{"1min":23.66,"5min":208.93799999999999,"15min":208.93799999999999},"last":9.2370000000000001},{"interface":"front","average":{"1min":13.587,"5min":16.247,"15min":24.045999999999999},"min":{"1min":7.3310000000000004,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":24.765000000000001,"5min":208.57400000000001,"15min":208.57400000000001},"last":8.3369999999999997}]},{"osd":3,"last update":"Wed Dec 6 09:27:10 2023","interfaces":[{"interface":"back","average":{"1min":17.373999999999999,"5min":15.544,"15min":17.196999999999999},"min":{"1min":10.898999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":25.806999999999999,"5min":43.276000000000003,"15min":43.276000000000003},"last":17.100999999999999},{"interface":"front","average":{"1min":9.9079999999999995,"5min":10.916,"15min":14.124000000000001},"min":{"1min":6.4320000000000004,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":17.137,"5min":41.493000000000002,"15min":41.493000000000002},"last":6.9080000000000004}]},{"osd":4,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":16.132000000000001,"5min":16.742999999999999,"15min":22.754999999999999},"min":{"1min":9.4000000000000004,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":27.443999999999999,"5min":117.10299999999999,"15min":117.10299999999999},"last":18.021000000000001},{"interface":"front","average":{"1min":18.768000000000001,"5min":18.789000000000001,"15min":25.478000000000002},"min":{"1min":8.4570000000000007,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":29.385000000000002,"5min":111.874,"15min":111.874},"last":16.210000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:27:00 2023","interfaces":[{"interface":"back","average":{"1min":12.522,"5min":12.273,"15min":16.116},"min":{"1min":7.3929999999999998,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":22.010000000000002,"5min":39.886000000000003,"15min":39.886000000000003},"last":10.478999999999999},{"interface":"front","average":{"1min":17.369,"5min":17.016999999999999,"15min":21.588999999999999},"min":{"1min":11.013999999999999,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":30.199999999999999,"5min":60.530999999999999,"15min":60.530999999999999},"last":20.442}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":20.678999999999998,"5min":20.678999999999998,"15min":20.678999999999998},"min":{"1min":13.064,"5min":13.064,"15min":13.064},"max":{"1min":31.757999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":18.939},{"interface":"front","average":{"1min":15.526999999999999,"5min":15.526999999999999,"15min":15.526999999999999},"min":{"1min":9.6890000000000001,"5min":9.6890000000000001,"15min":9.6890000000000001},"max":{"1min":28.809000000000001,"5min":28.809000000000001,"15min":28.809000000000001},"last":12.446}]}]},{"osd":5,"up_from":14,"seq":60129542217,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":643339,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":9.0600000000000005,"5min":10.461,"15min":13.067},"min":{"1min":4.5670000000000002,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":18.443000000000001,"5min":45.491,"15min":45.491},"last":4.7530000000000001},{"interface":"front","average":{"1min":8.7940000000000005,"5min":10.765000000000001,"15min":12.897},"min":{"1min":4.4349999999999996,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":14.475,"5min":49.991999999999997,"15min":49.991999999999997},"last":7.5289999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":11.146000000000001,"5min":13.614000000000001,"15min":21.236000000000001},"min":{"1min":5.6900000000000004,"5min":4.6710000000000003,"15min":4.6710000000000003},"max":{"1min":18.754000000000001,"5min":220.13900000000001,"15min":220.13900000000001},"last":13.528},{"interface":"front","average":{"1min":10.789,"5min":17.363,"15min":29.614999999999998},"min":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":17.491,"5min":344.79599999999999,"15min":344.79599999999999},"last":9.3339999999999996}]},{"osd":2,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.129,"5min":12.884,"15min":13.379},"min":{"1min":9.1910000000000007,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":24.969999999999999,"5min":53.235999999999997,"15min":53.235999999999997},"last":12.98},{"interface":"front","average":{"1min":14.458,"5min":13.545999999999999,"15min":15.108000000000001},"min":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":26.155000000000001,"5min":54.747,"15min":54.747},"last":8.4130000000000003}]},{"osd":3,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":14.762,"5min":14.917,"15min":13.975},"min":{"1min":9.6069999999999993,"5min":6.4189999999999996,"15min":6.4189999999999996},"max":{"1min":24.18,"5min":55.310000000000002,"15min":55.310000000000002},"last":14.596},{"interface":"front","average":{"1min":11.239000000000001,"5min":12.801,"15min":16.099},"min":{"1min":6.9889999999999999,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":19.765000000000001,"5min":51.183,"15min":51.183},"last":10.991}]},{"osd":4,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":12.808,"5min":14.959,"15min":19.792999999999999},"min":{"1min":9.0570000000000004,"5min":6.085,"15min":6.085},"max":{"1min":21.306999999999999,"5min":281.608,"15min":281.608},"last":12.676},{"interface":"front","average":{"1min":16.542999999999999,"5min":16.850999999999999,"15min":21.760999999999999},"min":{"1min":11.683999999999999,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":23.295999999999999,"5min":225.83600000000001,"15min":225.83600000000001},"last":17.292000000000002}]},{"osd":6,"last update":"Wed Dec 6 09:27:05 2023","interfaces":[{"interface":"back","average":{"1min":16.638999999999999,"5min":16.809000000000001,"15min":20.645},"min":{"1min":11.105,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":22.853000000000002,"5min":198.667,"15min":198.667},"last":18.283999999999999},{"interface":"front","average":{"1min":16.501000000000001,"5min":17.471,"15min":22.067},"min":{"1min":10.128,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":24.225000000000001,"5min":208.446,"15min":208.446},"last":15.842000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:26:57 2023","interfaces":[{"interface":"back","average":{"1min":14.349,"5min":14.349,"15min":14.349},"min":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":9.9410000000000007},{"interface":"front","average":{"1min":18.501000000000001,"5min":18.501000000000001,"15min":18.501000000000001},"min":{"1min":12.881,"5min":12.881,"15min":12.881},"max":{"1min":25.683,"5min":25.683,"15min":25.683},"last":15.212999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542217,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":733184,"data_stored":647785,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":8.8759999999999994,"5min":8.9649999999999999,"15min":10.57},"min":{"1min":5.6459999999999999,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":26.015000000000001,"5min":40.225999999999999,"15min":40.225999999999999},"last":11.645},{"interface":"front","average":{"1min":11.209,"5min":10.295999999999999,"15min":11.488},"min":{"1min":5.8639999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":20.613,"5min":30.405000000000001,"15min":30.405000000000001},"last":10.731}]},{"osd":2,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":10.425000000000001,"5min":13.679,"15min":11.686},"min":{"1min":7.0549999999999997,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":21.376000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":9.5649999999999995},{"interface":"front","average":{"1min":15.489000000000001,"5min":17.225999999999999,"15min":16.045999999999999},"min":{"1min":10.004,"5min":6.415,"15min":6.415},"max":{"1min":30.391999999999999,"5min":164.911,"15min":164.911},"last":16.396999999999998}]},{"osd":3,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.678000000000001,"5min":14.433999999999999,"15min":19.654},"min":{"1min":5.359,"5min":5.359,"15min":5.359},"max":{"1min":20.265999999999998,"5min":122.762,"15min":122.762},"last":28.663},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.653,"15min":19.021000000000001},"min":{"1min":7.1059999999999999,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":22.109000000000002,"5min":129.99100000000001,"15min":129.99100000000001},"last":25.053999999999998}]},{"osd":4,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.468999999999999,"5min":18.315000000000001,"15min":27.951000000000001},"min":{"1min":9.1140000000000008,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":49.097000000000001,"5min":359.46300000000002,"15min":359.46300000000002},"last":9.1859999999999999},{"interface":"front","average":{"1min":15.288,"5min":17.658999999999999,"15min":26.722999999999999},"min":{"1min":6.4539999999999997,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":47.192,"5min":314.47699999999998,"15min":314.47699999999998},"last":6.7389999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":19.167999999999999,"5min":21.059999999999999,"15min":25.846},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":81.923000000000002,"5min":152.97399999999999,"15min":152.97399999999999},"last":13.445},{"interface":"front","average":{"1min":16.448,"5min":18.698,"15min":24.085000000000001},"min":{"1min":10.108000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.151000000000003,"5min":154.53100000000001,"15min":154.53100000000001},"last":12.574999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:27:06 2023","interfaces":[{"interface":"back","average":{"1min":14.266999999999999,"5min":15.220000000000001,"15min":17.032},"min":{"1min":8.218,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":20.634,"5min":43.045000000000002,"15min":43.045000000000002},"last":17.545000000000002},{"interface":"front","average":{"1min":16.388000000000002,"5min":17.977,"15min":20.286999999999999},"min":{"1min":11.009,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":27.966999999999999,"5min":41.356000000000002,"15min":41.356000000000002},"last":18.413}]},{"osd":7,"last update":"Wed Dec 6 09:26:56 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":19.687999999999999,"15min":19.687999999999999},"min":{"1min":11.547000000000001,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":32.191000000000003,"5min":32.191000000000003,"15min":32.191000000000003},"last":15.148},{"interface":"front","average":{"1min":19.468,"5min":19.468,"15min":19.468},"min":{"1min":13.093,"5min":13.093,"15min":13.093},"max":{"1min":36.454000000000001,"5min":36.454000000000001,"15min":36.454000000000001},"last":14.021000000000001}]}]},{"osd":4,"up_from":0,"seq":55834574923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:27:52.698 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-06T09:27:52.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:27:52.779 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:52.793+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:52.793 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:52.793 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 337.83030280700001, 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:52.794 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:52.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:52.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:52.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:52.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:52.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:52.800 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.800 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:52.801 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.801 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:52.801 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:52.801 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:52.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:52.880 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:52.894+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:52.934 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:52.935 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:52.935 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:52.935 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:52.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:52.958 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:52.958 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:52.980 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:52.994+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:53.044 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:53.044 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:53.073 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:53.073 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:53.073 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:53.074 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:53.074 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:53.081 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:53.095+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:53.082 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:53.170 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:53.171 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:53.182 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:53.196+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:53.254 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:53.255 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:53.255 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:53.255 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:53.255 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:53.264 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:53.279 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:53.280 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:53.284 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:53.297+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:53.354 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":46,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:27:53.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:27:53.382 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:53.383 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:53.384 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:53.398+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:53.487 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:53.500+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:53.499 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:53.499 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:53.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:53.579 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:53.588 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:53.602+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:53.670 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:53.671 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:53.678 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:53.678 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:53.687 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:53.701+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:53.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:53.788 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:53.802+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:53.858 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:53.858 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:53.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:53.888 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:53.902+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:53.991 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:54.004+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:54.001 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.001 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:54.091 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:54.105+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:54.140 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.140 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:54.192 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:54.206+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:54.274 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.274 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:54.293 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:54.307+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:54.394 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:54.408+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:54.466 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.466 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.475 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:54.518 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:54.532+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:54.596 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:54.611+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:54.602 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.602 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:54.698 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:54.712+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:54.783 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.783 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:54.833 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:54.847+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:54.899 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:54.913+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:54.900 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:27:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:54.936 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:55.021 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:55.035+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:55.102 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:55.115+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:55.126 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:55.127 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:55.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:55.221 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:55.235+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:55.303 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:55.317+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:55.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:55.404 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:55.418+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:55.485 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:55.485 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:55.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:55.504 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:55.518+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:55.609 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:55.623+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:55.623 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:55.623 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:55.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:27:55.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T09:27:55.717 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:55.730+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:55.804 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:55.804 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:55.807 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:55.821+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:55.813 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:27:55.909 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:55.922+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:55.981 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:55.981 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:55.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:27:55.995 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:56.009 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:56.023+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:56.105 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:56.105 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:56.110 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:56.124+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:56.148 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:56.148 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:56.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:27:56.207 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:56.207 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:56.211 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:56.225+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:56.312 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:56.329 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:56.343+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:56.355 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:56.355 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:56.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:56.355 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:56.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.355 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 205.71262012, 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.203558082, 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646072+0000", 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.646088+0000", 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6297999999999999e-05 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.670066+0000", 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023978154000000002 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.677815+0000", 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0077484729999999996 2023-12-06T09:27:56.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.680864+0000", 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030490819999999998 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.692390+0000", 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011526563 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.695994+0000", 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036032339999999999 2023-12-06T09:27:56.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.849630+0000", 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15363627799999999 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 205.40739881900001, 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030764554, 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2023-12-06T09:27:56.359 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951293+0000", 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.951305+0000", 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1775e-05 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955091+0000", 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037857199999999998 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955437+0000", 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00034590800000000002 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:56.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.955858+0000", 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00042182800000000001 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.957808+0000", 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001949737 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.958121+0000", 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313178 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:56.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:24:30.982058+0000", 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.023936407999999999 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:12 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 40.523661027000003, 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029206599999999998, 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:56.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 12 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835031+0000", 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:56.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835047+0000", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5968999999999999e-05 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835217+0000", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017016899999999999 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835416+0000", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000198677 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.835585+0000", 2023-12-06T09:27:56.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016903299999999999 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.836489+0000", 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00090463399999999997 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:27:15.837952+0000", 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001462178 2023-12-06T09:27:56.366 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.367 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:56.367 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.367 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.367 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:56.367 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:56.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:27:56.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:27:56.405 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:56.449 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:56.463+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:56.490 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:56.517 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:56.531+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:56.538 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:56.538 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:56.539 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2023-12-06T09:27:56.539 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:56.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:27:56.595 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:56.595 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:56.623 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:56.637+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:56.682 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:56.682 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:56.716 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:56.730+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:56.718 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:56.718 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2023-12-06T09:27:56.718 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2023-12-06T09:27:56.718 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2023-12-06T09:27:56.718 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:56.728 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:27:56.817 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:56.831+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:56.900 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.4165.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 341.93926562500002, 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092802343999999995, 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.4165", 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:56.901 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 1 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974086+0000", 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2023-12-06T09:27:56.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.974314+0000", 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00022784300000000001 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.978894+0000", 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045802239999999999 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.983168+0000", 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042738129999999996 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:14.991844+0000", 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008676352 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.011299+0000", 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019454618999999999 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2023-12-06T09:22:15.066888+0000", 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055589492999999997 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:56.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.905 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-12-06T09:27:56.905 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-12-06T09:27:56.905 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:27:56.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:27:56.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:27:56.954 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:56.968+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:57.018 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:57.032+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:57.042 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:57.042 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:57.042 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:57.042 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:57.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:27:57.086 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":49,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:27:57.135 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:57.149+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:57.225 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:57.239+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:57.238 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:57.238 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:57.238 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:57.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:27:57.331 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:57.345+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:57.419 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:57.419 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:57.419 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:57.419 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:57.419 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:57.421 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:57.435+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:57.428 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:27:57.522 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:57.535+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:57.598 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:57.598 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:57.598 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:57.598 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:57.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:27:57.623 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:57.637+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:57.699 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 7, 0, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T09:27:57.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T09:27:57.699 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 3, 6, 7, 0, 5] 2023-12-06T09:27:57.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2023-12-06T09:27:57.724 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:57.738+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:57.777 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:57.777 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:57.777 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:57.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:27:57.825 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:57.839+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:57.926 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:57.940+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:57.995 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:57.995 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:57.995 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:57.995 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:1 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 359.288249065, 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.16287976600000001, 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:57.996 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680088+0000", 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.680260+0000", 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017237700000000001 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.684470+0000", 2023-12-06T09:27:57.997 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0042094929999999999 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.685342+0000", 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00087208000000000001 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.693799+0000", 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0084574769999999997 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:57.998 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.720187+0000", 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026387913999999998 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.729449+0000", 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092617399999999992 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.829956+0000", 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.10050671899999999 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:57.999 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:58.842968+0000", 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.013011966 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:20 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.59379519599997, 2023-12-06T09:27:58.000 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.133273686, 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.001 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374542+0000", 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.374551+0000", 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4029999999999996e-06 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379026+0000", 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0044749860000000002 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.002 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379174+0000", 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014775199999999999 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.379289+0000", 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011478500000000001 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.003 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380431+0000", 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001142557 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.380811+0000", 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037974500000000002 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507156+0000", 2023-12-06T09:27:58.004 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.126345283 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.507815+0000", 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065917500000000002 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:30 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'16, mlcod=17'14)", 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.15090271999998, 2023-12-06T09:27:58.005 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.049760041999999997, 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817434+0000", 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.006 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817444+0000", 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.4350000000000003e-06 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817593+0000", 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014920499999999999 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817762+0000", 2023-12-06T09:27:58.007 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000169285 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.817878+0000", 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011589799999999999 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.818928+0000", 2023-12-06T09:27:58.008 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010496100000000001 2023-12-06T09:27:58.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.009 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.009 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.009 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.819356+0000", 2023-12-06T09:27:58.009 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00042865900000000002 2023-12-06T09:27:58.009 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.866671+0000", 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.047314637999999999 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.867194+0000", 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052331199999999995 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.010 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:31 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.14207643899999, 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.056917051000000003, 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.011 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826260+0000", 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.012 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826268+0000", 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.0399999999999993e-06 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826418+0000", 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014939900000000001 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.013 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826540+0000", 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012250100000000001 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826660+0000", 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119988 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.827698+0000", 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0010379849999999999 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.014 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.828061+0000", 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000362822 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.882117+0000", 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054056178000000003 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.883177+0000", 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001060138 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.015 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.14145476700003, 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.058543023999999999, 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.016 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826882+0000", 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.826888+0000", 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.8989999999999998e-06 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.832950+0000", 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.006062558 2023-12-06T09:27:58.017 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833059+0000", 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000108684 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.833164+0000", 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104539 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837500+0000", 2023-12-06T09:27:58.018 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0043360209999999998 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.837855+0000", 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035541099999999998 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.884515+0000", 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.046659995000000003 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.019 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885425+0000", 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000909917 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.13350828599999, 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051197670000000001, 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.020 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834829+0000", 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834835+0000", 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3380000000000001e-06 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.834930+0000", 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.5285999999999995e-05 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835041+0000", 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011045 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.022 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835144+0000", 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00010323300000000001 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845298+0000", 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010154470000000001 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.845643+0000", 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00034508299999999999 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.885454+0000", 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039810393999999999 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886026+0000", 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00057241599999999996 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.024 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.13303426800002, 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051282057999999998, 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.025 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835303+0000", 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835308+0000", 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.5659999999999997e-06 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835407+0000", 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.8858999999999997e-05 2023-12-06T09:27:58.026 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835512+0000", 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000104677 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835604+0000", 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2650000000000002e-05 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.851858+0000", 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01625352 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.852232+0000", 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037396599999999999 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886053+0000", 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033821395999999997 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.028 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886585+0000", 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053142399999999996 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 358.13257991299997, 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.051323981999999997, 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.029 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835757+0000", 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.030 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.835762+0000", 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.4210000000000003e-06 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860133+0000", 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024371179999999999 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860280+0000", 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0001461 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.860401+0000", 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012136100000000001 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862511+0000", 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0021101729999999999 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.862865+0000", 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035340399999999998 2023-12-06T09:27:58.032 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.886604+0000", 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023739501 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:21:59.887081+0000", 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000476842 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.033 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:53 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'31, mlcod=18'29)", 2023-12-06T09:27:58.034 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:58.034 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 357.74858766199998, 2023-12-06T09:27:58.034 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.035157368000000001, 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219749+0000", 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.035 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219758+0000", 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.7479999999999999e-06 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.219890+0000", 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000132509 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220043+0000", 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015301099999999999 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.036 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.220158+0000", 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000115016 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221415+0000", 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001256666 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.037 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.221800+0000", 2023-12-06T09:27:58.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00038492799999999998 2023-12-06T09:27:58.038 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.038 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.038 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.038 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254254+0000", 2023-12-06T09:27:58.038 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032453967 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.254907+0000", 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00065252300000000001 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.039 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:71 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'45, mlcod=18'43)", 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 357.36343825699998, 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044624416, 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.040 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604899+0000", 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.604908+0000", 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.0589999999999992e-06 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605752+0000", 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00084467500000000003 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.041 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.605887+0000", 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013516299999999999 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606001+0000", 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011319699999999999 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.606821+0000", 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00082008799999999996 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.042 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.607231+0000", 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000410596 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.648706+0000", 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041474529000000003 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.649523+0000", 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000817109 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:72 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'46, mlcod=18'43)", 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 357.35116294599999, 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034367897000000001, 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.044 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617174+0000", 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.617182+0000", 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.3760000000000008e-06 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618230+0000", 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001047568 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.045 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618349+0000", 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000119245 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618453+0000", 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000103878 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619422+0000", 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00096927299999999999 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.046 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.619785+0000", 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000363087 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.650961+0000", 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.031175220999999999 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651542+0000", 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00058124900000000005 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.047 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:73 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'47, mlcod=18'43)", 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 357.34967736800002, 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033526317, 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.048 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618659+0000", 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.618665+0000", 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 5.7019999999999998e-06 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.049 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630550+0000", 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.011885305000000001 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630699+0000", 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014831800000000001 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.630814+0000", 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011509 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632208+0000", 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001393725 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.632609+0000", 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00040173500000000002 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.651651+0000", 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01904148 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.051 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:00.652186+0000", 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00053496200000000005 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:104 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'66, mlcod=18'64)", 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.44314021000002, 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.041445835, 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.052 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525197+0000", 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525206+0000", 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.2320000000000007e-06 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.053 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525389+0000", 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018282699999999999 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525529+0000", 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014044200000000001 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.525643+0000", 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011381400000000001 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.054 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526528+0000", 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00088461799999999999 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.526938+0000", 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041025600000000001 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566034+0000", 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.039095893 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.055 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.566642+0000", 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00060875299999999996 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:105 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=18'64)", 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.43800790699999, 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044813763999999999, 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.056 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530329+0000", 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530337+0000", 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4659999999999999e-06 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530469+0000", 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00013117200000000001 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530589+0000", 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000120221 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.530688+0000", 2023-12-06T09:27:58.058 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.9072000000000003e-05 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538871+0000", 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0081833050000000001 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.539288+0000", 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00041667499999999998 2023-12-06T09:27:58.059 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.574101+0000", 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.034812880999999997 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575143+0000", 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001041972 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.060 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:106 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=18'64)", 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.437181877, 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044752753999999999, 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:58.061 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531155+0000", 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.531161+0000", 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.3400000000000003e-06 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.062 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533171+0000", 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0020099760000000001 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533321+0000", 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015001500000000001 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533475+0000", 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015348799999999999 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.063 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.542705+0000", 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0092299779999999998 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.543058+0000", 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035351300000000002 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575184+0000", 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.032125314000000002 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.064 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575908+0000", 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00072413000000000002 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:107 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=18'64)", 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.43461241599999, 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043002805999999998, 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.065 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533724+0000", 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533733+0000", 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2260000000000005e-06 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.066 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.533873+0000", 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00014063500000000001 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534035+0000", 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016212299999999999 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534213+0000", 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00017791000000000001 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.067 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548290+0000", 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.014076627 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.548665+0000", 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00037471599999999998 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.068 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.575931+0000", 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.027266117999999999 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576727+0000", 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00079645099999999999 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:108 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=18'64)", 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:58.069 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.43386694600002, 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.043118055000000002, 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534470+0000", 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.070 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534478+0000", 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.2579999999999995e-06 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534635+0000", 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00015728699999999999 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534825+0000", 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00018975599999999999 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.534986+0000", 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00016050400000000001 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.551676+0000", 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.016690335000000001 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.552027+0000", 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000350589 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.072 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.576757+0000", 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.024730693000000002 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.577588+0000", 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000830633 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:109 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=18'64)", 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:58.073 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.43306087500002, 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.044320379, 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535276+0000", 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.074 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.535284+0000", 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4880000000000007e-06 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.538419+0000", 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0031342610000000002 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546038+0000", 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0076193249999999997 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.075 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546160+0000", 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00012243700000000001 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555051+0000", 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0088909560000000002 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.555405+0000", 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00035377199999999997 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.076 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579051+0000", 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.023645457000000002 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579596+0000", 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00054568300000000004 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_repop(client.4139.0:110 1.0 e21/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=18'64)", 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:58.077 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 349.42195642299998, 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.033923153999999997, 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546380+0000", 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.078 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546387+0000", 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 6.8569999999999997e-06 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546488+0000", 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.000100914 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546599+0000", 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00011130500000000001 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.079 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.546693+0000", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 9.3980000000000005e-05 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567109+0000", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.020415578 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.567634+0000", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00052513799999999997 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "commit_sent", 2023-12-06T09:27:58.080 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.579618+0000", 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.01198406 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:22:08.580304+0000", 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00068532200000000004 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 200.648624286, 2023-12-06T09:27:58.081 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.095926704000000002, 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "started", 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319713+0000", 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.082 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.319721+0000", 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 8.4889999999999995e-06 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.346094+0000", 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.026372639999999999 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.353690+0000", 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00759663 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.083 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.354398+0000", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00070748399999999998 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.359769+0000", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0053715530000000003 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "started", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.361455+0000", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001686108 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:24:37.415639+0000", 2023-12-06T09:27:58.084 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.054183799999999997 2023-12-06T09:27:58.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.085 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.085 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.085 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.085 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:58.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:27:58.087 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:27:58.100+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:27:58.187 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:58.201+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:27:58.263 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:58.263 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:58.263 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:58.263 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:58.274 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:27:58.318 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:58.332+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:58.323 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:27:58.336+0000 ff5f640 -1 mon.a@0(leader).osd e103 definitely_dead 0 2023-12-06T09:27:58.389 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:58.403+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:58.407 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:58.408 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:58.408 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:58.408 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:58.408 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:58.416 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:27:58.490 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:58.504+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:58.587 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [ 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "description": "osd_op(client.4165.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "initiated_at": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "age": 43.378911727000002, 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.010005948000000001, 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "type_data": { 2023-12-06T09:27:58.588 INFO:teuthology.orchestra.run.smithi132.stdout: "flag_point": "reached pg", 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "client_info": { 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "client": "client.4165", 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "client_addr": "172.21.15.88:0/5809301", 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "tid": 11 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "events": [ 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "initiated", 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.589 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "header_read", 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221294+0000", 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "throttled", 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221310+0000", 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 1.6829000000000001e-05 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "all_read", 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221505+0000", 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00019430599999999999 2023-12-06T09:27:58.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "dispatched", 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.221716+0000", 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.00021168399999999999 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "queued_for_pg", 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.224250+0000", 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0025338729999999999 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.591 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "reached_pg", 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.229715+0000", 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.0054648589999999999 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: { 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: "event": "done", 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: "time": "2023-12-06T09:27:15.231300+0000", 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 0.001584397 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2023-12-06T09:27:58.593 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2023-12-06T09:27:58.593 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:58.594 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:58.608+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:58.597 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:27:58.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T09:27:58.696 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:58.709+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:58.765 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:58.766 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:58.766 INFO:teuthology.orchestra.run.smithi132.stdout: "num_ops": 0 2023-12-06T09:27:58.766 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:58.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:27:58.796 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:58.811+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:58.831 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:17.177 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:58.890 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:17.236 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:58.898 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:58.912+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:58.901 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:58.901 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [], 2023-12-06T09:27:58.901 INFO:teuthology.orchestra.run.smithi132.stdout: "complaint_time": 30, 2023-12-06T09:27:58.901 INFO:teuthology.orchestra.run.smithi132.stdout: "num_blocked_ops": 0 2023-12-06T09:27:58.901 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:58.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:27:58.936 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0: {} 2023-12-06T09:27:58.936 INFO:teuthology.orchestra.run.smithi088.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T09:27:59.002 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:59.013+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:59.024 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1: {} 2023-12-06T09:27:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T09:27:59.078 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2023-12-06T09:27:59.078 INFO:teuthology.orchestra.run.smithi132.stdout: "size": 20, 2023-12-06T09:27:59.078 INFO:teuthology.orchestra.run.smithi132.stdout: "duration": 600, 2023-12-06T09:27:59.078 INFO:teuthology.orchestra.run.smithi132.stdout: "ops": [] 2023-12-06T09:27:59.078 INFO:teuthology.orchestra.run.smithi132.stdout:} 2023-12-06T09:27:59.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T09:27:59.112 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2: {} 2023-12-06T09:27:59.113 INFO:teuthology.orchestra.run.smithi088.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T09:27:59.128 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:59.142+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:59.150 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:17.496 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:27:59.202 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:27:59.216+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:27:59.207 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3: {} 2023-12-06T09:27:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T09:27:59.225 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:59.225 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:59.234 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T09:27:59.282 INFO:teuthology.orchestra.run.smithi088.stderr:marked out osd.2. 2023-12-06T09:27:59.302 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:27:59.316+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:27:59.318 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4: {} 2023-12-06T09:27:59.318 INFO:teuthology.orchestra.run.smithi088.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T09:27:59.403 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:59.417+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:59.435 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:59.435 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:59.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T09:27:59.462 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5: {} 2023-12-06T09:27:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T09:27:59.506 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:27:59.520+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:27:59.564 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6: {} 2023-12-06T09:27:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T09:27:59.604 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:27:59.618+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:27:59.617 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:59.617 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:59.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T09:27:59.663 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7: {} 2023-12-06T09:27:59.663 INFO:teuthology.orchestra.run.smithi088.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T09:27:59.706 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:27:59.720+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:27:59.799 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:59.799 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:59.806 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:27:59.820+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:27:59.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T09:27:59.907 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:27:59.921+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:27:59.936 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:27:59.937 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:27:59.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T09:28:00.008 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:28:00.022+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:28:00.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T09:28:00.123 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:00.123 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:00.124 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:28:00.137+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:28:00.133 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T09:28:00.218 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:28:00.232+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:28:00.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T09:28:00.311 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:28:00.325+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:28:00.316 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:00.316 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:00.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T09:28:00.412 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:28:00.426+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:28:00.506 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:00.506 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:00.514 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:28:00.528+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:28:00.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T09:28:00.615 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:28:00.629+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:28:00.684 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:00.684 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:00.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T09:28:00.716 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:28:00.730+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:28:00.804 INFO:teuthology.orchestra.run.smithi088.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","modified":"2023-12-06T09:20:12.041363Z","created":"2023-12-06T09:20:12.041363Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3301","nonce":0},{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T09:28:00.817 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:28:00.831+0000 ac7d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:28:00.821 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T09:28:00.821 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2023-12-06T09:28:00.831 INFO:tasks.thrashosds:joining thrashosds 2023-12-06T09:28:00.873 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:00.873 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:00.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T09:28:01.016 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.016 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.026 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T09:28:01.154 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.155 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T09:28:01.292 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.293 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.301 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T09:28:01.441 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.441 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.450 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T09:28:01.580 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.581 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T09:28:01.720 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.720 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T09:28:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is set 2023-12-06T09:28:01.870 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:01.870 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:01.880 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T09:28:02.016 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.016 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:02.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T09:28:02.155 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.155 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:02.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T09:28:02.290 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.290 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:02.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T09:28:02.426 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.426 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:02.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T09:28:02.567 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.567 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:02.576 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T09:28:02.706 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.706 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:02.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T09:28:02.867 INFO:teuthology.orchestra.run.smithi132.stdout:ERROR: (22) Invalid argument 2023-12-06T09:28:02.867 INFO:teuthology.orchestra.run.smithi132.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T09:28:03.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T09:28:04.299 INFO:tasks.thrashosds.thrasher:inning osd 2023-12-06T09:28:04.300 INFO:tasks.thrashosds.thrasher:Adding osd 2 2023-12-06T09:28:04.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2023-12-06T09:28:04.884 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:28:04.895+0000 ff5f640 -1 mon.a@0(leader).osd e106 definitely_dead 0 2023-12-06T09:28:05.669 INFO:teuthology.orchestra.run.smithi088.stderr:noscrub is unset 2023-12-06T09:28:05.672 INFO:teuthology.orchestra.run.smithi088.stderr:marked in osd.2. 2023-12-06T09:28:05.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T09:28:05.687 INFO:tasks.thrashosds.thrasher:Added osd 2 2023-12-06T09:28:05.687 INFO:tasks.thrashosds.thrasher:inning osd 2023-12-06T09:28:05.687 INFO:tasks.thrashosds.thrasher:Adding osd 4 2023-12-06T09:28:05.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2023-12-06T09:28:06.552 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:28:06.562+0000 ff5f640 -1 mon.a@0(leader).osd e107 definitely_dead 0 2023-12-06T09:28:06.837 INFO:teuthology.orchestra.run.smithi088.stderr:marked in osd.4. 2023-12-06T09:28:06.842 INFO:teuthology.orchestra.run.smithi088.stderr:nodeep-scrub is unset 2023-12-06T09:28:06.856 INFO:tasks.thrashosds.thrasher:Added osd 4 2023-12-06T09:28:06.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2023-12-06T09:28:07.140 INFO:teuthology.orchestra.run.smithi088.stdout:mon.a: {} 2023-12-06T09:28:07.140 INFO:teuthology.orchestra.run.smithi088.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T09:28:07.256 INFO:teuthology.orchestra.run.smithi088.stdout:mon.b: {} 2023-12-06T09:28:07.256 INFO:teuthology.orchestra.run.smithi088.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T09:28:07.388 INFO:teuthology.orchestra.run.smithi088.stdout:mon.c: {} 2023-12-06T09:28:07.388 INFO:teuthology.orchestra.run.smithi088.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T09:28:07.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-06T09:28:08.323 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.0 to 1 (10000) 2023-12-06T09:28:08.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2023-12-06T09:28:08.339 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:46.318 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.387 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:40.701 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.387 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:40.702 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.391 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:46.369 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.418 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:01:40.733 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.431 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:46.409 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.463 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:26.808 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.468 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:26.814 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:08.491 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:26.837 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:10.085 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.0 primary-affinity to 1 (8655362) 2023-12-06T09:28:10.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-06T09:28:11.309 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.1 to 1 (10000) 2023-12-06T09:28:11.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2023-12-06T09:28:12.325 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.1 primary-affinity to 1 (8655362) 2023-12-06T09:28:12.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-06T09:28:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.2 to 1 (10000) 2023-12-06T09:28:13.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2023-12-06T09:28:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.2 primary-affinity to 1 (8655362) 2023-12-06T09:28:14.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-06T09:28:15.639 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.3 to 1 (10000) 2023-12-06T09:28:15.653 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2023-12-06T09:28:16.744 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.3 primary-affinity to 1 (8655362) 2023-12-06T09:28:16.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-12-06T09:28:17.948 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.4 to 1 (10000) 2023-12-06T09:28:17.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2023-12-06T09:28:19.634 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.4 primary-affinity to 1 (8655362) 2023-12-06T09:28:19.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-12-06T09:28:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.5 to 1 (10000) 2023-12-06T09:28:20.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2023-12-06T09:28:21.711 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.5 primary-affinity to 1 (8655362) 2023-12-06T09:28:21.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-12-06T09:28:22.814 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.6 to 1 (10000) 2023-12-06T09:28:22.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2023-12-06T09:28:24.322 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.6 primary-affinity to 1 (8655362) 2023-12-06T09:28:24.336 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-12-06T09:28:25.952 INFO:teuthology.orchestra.run.smithi088.stderr:reweighted osd.7 to 1 (10000) 2023-12-06T09:28:25.966 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2023-12-06T09:28:26.965 INFO:teuthology.orchestra.run.smithi088.stderr:set osd.7 primary-affinity to 1 (8655362) 2023-12-06T09:28:26.979 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2023-12-06T09:28:26.980 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2023-12-06T09:28:26.980 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2023-12-06T09:28:26.980 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2023-12-06T09:28:26.980 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-06T09:28:26.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:28:27.656 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:27.656 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":126,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:28:26.904433+0000","last_up_change":"2023-12-06T09:25:55.041559+0000","last_in_change":"2023-12-06T09:28:06.637886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T09:22:01.832571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":3.5,"raw_score_stable":3.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":51,"up_from":60,"up_thru":104,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1573021680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":1573021680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":1573021680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1573021680}]},"public_addr":"172.21.15.132:6812/1573021680","cluster_addr":"172.21.15.132:6813/1573021680","heartbeat_back_addr":"172.21.15.132:6815/1573021680","heartbeat_front_addr":"172.21.15.132:6814/1573021680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":51}],"pg_upmap":[{"pgid":"2.4","osds":[2,5]},{"pgid":"2.7","osds":[7,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:28:27.672 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-06T09:28:27.672 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:28:28.355 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:28.355 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:28:28.259885+0000","last_up_change":"2023-12-06T09:25:55.041559+0000","last_in_change":"2023-12-06T09:28:06.637886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T09:22:01.832571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":3.5,"raw_score_stable":3.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":51,"up_from":60,"up_thru":104,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1573021680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":1573021680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":1573021680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1573021680}]},"public_addr":"172.21.15.132:6812/1573021680","cluster_addr":"172.21.15.132:6813/1573021680","heartbeat_back_addr":"172.21.15.132:6815/1573021680","heartbeat_front_addr":"172.21.15.132:6814/1573021680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":51}],"pg_upmap":[{"pgid":"2.4","osds":[2,5]},{"pgid":"2.7","osds":[7,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:28:28.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-06T09:28:28.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-06T09:28:28.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-06T09:28:28.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-06T09:28:28.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-06T09:28:28.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-06T09:28:28.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-06T09:28:28.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-06T09:28:28.387 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:06.365 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.391 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:06.370 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.433 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:46.778 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.436 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:00.751 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.437 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:00.752 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.437 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:46.780 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.465 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:00.777 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.475 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:06.454 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.505 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:46.851 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:28.857 INFO:teuthology.orchestra.run.smithi088.stdout:257698037793 2023-12-06T09:28:28.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T09:28:28.862 INFO:teuthology.orchestra.run.smithi088.stdout:60129542225 2023-12-06T09:28:28.862 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T09:28:28.874 INFO:teuthology.orchestra.run.smithi088.stdout:64424509522 2023-12-06T09:28:28.875 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T09:28:28.903 INFO:teuthology.orchestra.run.smithi088.stdout:60129542225 2023-12-06T09:28:28.903 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T09:28:28.943 INFO:teuthology.orchestra.run.smithi088.stdout:60129542225 2023-12-06T09:28:28.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T09:28:28.965 INFO:teuthology.orchestra.run.smithi088.stdout:64424509522 2023-12-06T09:28:28.966 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T09:28:29.011 INFO:teuthology.orchestra.run.smithi088.stdout:64424509523 2023-12-06T09:28:29.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T09:28:29.015 INFO:teuthology.orchestra.run.smithi088.stdout:55834574931 2023-12-06T09:28:29.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T09:28:29.814 INFO:teuthology.orchestra.run.smithi088.stdout:257698037792 2023-12-06T09:28:29.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 257698037793 got 257698037792 for osd.7 2023-12-06T09:28:29.915 INFO:teuthology.orchestra.run.smithi088.stdout:60129542224 2023-12-06T09:28:29.929 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542224 for osd.6 2023-12-06T09:28:30.013 INFO:teuthology.orchestra.run.smithi088.stdout:64424509521 2023-12-06T09:28:30.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509522 got 64424509521 for osd.2 2023-12-06T09:28:30.124 INFO:teuthology.orchestra.run.smithi088.stdout:55834574930 2023-12-06T09:28:30.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574931 got 55834574930 for osd.4 2023-12-06T09:28:30.350 INFO:teuthology.orchestra.run.smithi088.stdout:60129542224 2023-12-06T09:28:30.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542224 for osd.5 2023-12-06T09:28:30.380 INFO:teuthology.orchestra.run.smithi088.stdout:64424509521 2023-12-06T09:28:30.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509522 got 64424509521 for osd.3 2023-12-06T09:28:30.448 INFO:teuthology.orchestra.run.smithi088.stdout:60129542224 2023-12-06T09:28:30.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542224 for osd.1 2023-12-06T09:28:30.647 INFO:teuthology.orchestra.run.smithi088.stdout:64424509522 2023-12-06T09:28:30.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509523 got 64424509522 for osd.0 2023-12-06T09:28:30.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T09:28:30.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T09:28:31.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T09:28:31.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T09:28:31.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T09:28:31.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T09:28:31.463 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T09:28:31.608 INFO:teuthology.orchestra.run.smithi088.stdout:257698037792 2023-12-06T09:28:31.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 257698037793 got 257698037792 for osd.7 2023-12-06T09:28:31.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T09:28:31.771 INFO:teuthology.orchestra.run.smithi088.stdout:60129542224 2023-12-06T09:28:31.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542224 for osd.6 2023-12-06T09:28:31.994 INFO:teuthology.orchestra.run.smithi088.stdout:64424509522 2023-12-06T09:28:32.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509522 got 64424509522 for osd.2 2023-12-06T09:28:32.009 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:32.118 INFO:teuthology.orchestra.run.smithi088.stdout:60129542226 2023-12-06T09:28:32.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542226 for osd.5 2023-12-06T09:28:32.135 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:32.146 INFO:teuthology.orchestra.run.smithi088.stdout:55834574932 2023-12-06T09:28:32.154 INFO:teuthology.orchestra.run.smithi088.stdout:64424509522 2023-12-06T09:28:32.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574931 got 55834574932 for osd.4 2023-12-06T09:28:32.162 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:32.171 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509522 got 64424509522 for osd.3 2023-12-06T09:28:32.171 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:32.262 INFO:teuthology.orchestra.run.smithi088.stdout:60129542226 2023-12-06T09:28:32.275 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542226 for osd.1 2023-12-06T09:28:32.276 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:32.284 INFO:teuthology.orchestra.run.smithi088.stdout:64424509523 2023-12-06T09:28:32.298 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509523 got 64424509523 for osd.0 2023-12-06T09:28:32.298 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:32.628 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T09:28:32.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T09:28:33.181 INFO:teuthology.orchestra.run.smithi088.stdout:257698037794 2023-12-06T09:28:33.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 257698037793 got 257698037794 for osd.7 2023-12-06T09:28:33.199 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:33.355 INFO:teuthology.orchestra.run.smithi088.stdout:60129542226 2023-12-06T09:28:33.368 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542225 got 60129542226 for osd.6 2023-12-06T09:28:33.368 DEBUG:teuthology.parallel:result is None 2023-12-06T09:28:33.368 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-06T09:28:33.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:28:33.921 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:33.922 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:28:33.934 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2023-12-06T09:28:31.986916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225320,"kb_used_data":8172,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749400,"statfs":{"total":773094113280,"available":772863385600,"internally_reserved":0,"allocated":8368128,"data_stored":3773896,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.066229"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.988825+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:28:26.988825+0000","last_peered":"2023-12-06T09:28:26.988825+0000","last_clean":"2023-12-06T09:28:26.988825+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:26.988825+0000","last_undegraded":"2023-12-06T09:28:26.988825+0000","last_fullsized":"2023-12-06T09:28:26.988825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.911896+0000","last_change":"2023-12-06T09:28:08.040199+0000","last_active":"2023-12-06T09:28:27.911896+0000","last_peered":"2023-12-06T09:28:27.911896+0000","last_clean":"2023-12-06T09:28:27.911896+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:27.911896+0000","last_undegraded":"2023-12-06T09:28:27.911896+0000","last_fullsized":"2023-12-06T09:28:27.911896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:42:47.793196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.734283+0000","last_change":"2023-12-06T09:28:07.998592+0000","last_active":"2023-12-06T09:28:27.734283+0000","last_peered":"2023-12-06T09:28:27.734283+0000","last_clean":"2023-12-06T09:28:27.734283+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:27.734283+0000","last_undegraded":"2023-12-06T09:28:27.734283+0000","last_fullsized":"2023-12-06T09:28:27.734283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:46:54.460756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T09:28:24.346168+0000","last_change":"2023-12-06T09:28:06.903121+0000","last_active":"2023-12-06T09:28:24.346168+0000","last_peered":"2023-12-06T09:28:24.346168+0000","last_clean":"2023-12-06T09:28:24.346168+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:24.346168+0000","last_undegraded":"2023-12-06T09:28:24.346168+0000","last_fullsized":"2023-12-06T09:28:24.346168+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:49:16.145368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.730818+0000","last_change":"2023-12-06T09:28:18.783833+0000","last_active":"2023-12-06T09:28:27.730818+0000","last_peered":"2023-12-06T09:28:27.730818+0000","last_clean":"2023-12-06T09:28:27.730818+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:27.730818+0000","last_undegraded":"2023-12-06T09:28:27.730818+0000","last_fullsized":"2023-12-06T09:28:27.730818+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:48:37.840608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.983941+0000","last_change":"2023-12-06T09:28:00.969260+0000","last_active":"2023-12-06T09:28:26.983941+0000","last_peered":"2023-12-06T09:28:26.983941+0000","last_clean":"2023-12-06T09:28:26.983941+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:26.983941+0000","last_undegraded":"2023-12-06T09:28:26.983941+0000","last_fullsized":"2023-12-06T09:28:26.983941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:54:26.793621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.736127+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:28:27.736127+0000","last_peered":"2023-12-06T09:28:27.736127+0000","last_clean":"2023-12-06T09:28:27.736127+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:27.736127+0000","last_undegraded":"2023-12-06T09:28:27.736127+0000","last_fullsized":"2023-12-06T09:28:27.736127+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.732589+0000","last_change":"2023-12-06T09:28:18.640184+0000","last_active":"2023-12-06T09:28:27.732589+0000","last_peered":"2023-12-06T09:28:27.732589+0000","last_clean":"2023-12-06T09:28:27.732589+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:27.732589+0000","last_undegraded":"2023-12-06T09:28:27.732589+0000","last_fullsized":"2023-12-06T09:28:27.732589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:06:02.343820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":243,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:28.786089+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:28:28.786089+0000","last_peered":"2023-12-06T09:28:28.786089+0000","last_clean":"2023-12-06T09:28:28.786089+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:28.786089+0000","last_undegraded":"2023-12-06T09:28:28.786089+0000","last_fullsized":"2023-12-06T09:28:28.786089+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":49.530000000000001},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":45.340000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":24.831},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.161000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":26.298999999999999},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":29.259}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":30.797999999999998},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":32.622}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":34.509999999999998},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":27.149000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":27.738},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":13.734999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:58 2023","interfaces":[{"interface":"back","average":{"1min":23.475000000000001,"5min":23.638999999999999,"15min":23.667000000000002},"min":{"1min":11.068,"5min":11.068,"15min":11.068},"max":{"1min":45.051000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":21.917000000000002},{"interface":"front","average":{"1min":22.222000000000001,"5min":22.856999999999999,"15min":22.963000000000001},"min":{"1min":13.550000000000001,"5min":11.01,"15min":11.01},"max":{"1min":41.414000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":21.184000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1028096,"data_stored":368992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":18.181000000000001},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":14.345000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":15.398},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":16.344999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":28.466000000000001},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":22.352}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":24.449999999999999},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":33.030000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":18.827999999999999},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":27.844999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":25.460999999999999},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":21.786000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":26.748999999999999},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":29.521000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":12.863},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":12.300000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":14.683},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":16.667999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":6.7539999999999996},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":16.056000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":8.8699999999999992},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":18.913}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":17.905000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":10.766}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":22.132999999999999},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":20.402999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":20.242999999999999,"5min":19.175000000000001,"15min":18.997},"min":{"1min":12.217000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":44.826000000000001,"5min":61.445999999999998,"15min":61.445999999999998},"last":13.907},{"interface":"front","average":{"1min":22.859000000000002,"5min":20.625,"15min":20.253},"min":{"1min":13.35,"5min":11.833,"15min":11.833},"max":{"1min":38.460000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":21.338999999999999}]}]},{"osd":7,"up_from":60,"seq":257698037794,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":4.0700000000000003},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":8.1699999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":18.030000000000001},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":13.34}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":7.0209999999999999},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":14.106999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":12.182},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":14.749000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":17.204999999999998},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":9.2629999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":10.500999999999999},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":15.430999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":18.806000000000001},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":11.301}]}]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":4.1959999999999997},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":7.343}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.141,"5min":12.663,"15min":14.898999999999999},"min":{"1min":7.5880000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":27.593,"5min":37.103999999999999,"15min":55.398000000000003},"last":15.6},{"interface":"front","average":{"1min":14.585000000000001,"5min":13.566000000000001,"15min":14.862},"min":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"max":{"1min":25.289000000000001,"5min":38.137,"15min":41.811999999999998},"last":12.631}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":8.1500000000000004},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":10.66}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":9.9789999999999992},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":12.015000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":19.532,"5min":15.497,"15min":22.34},"min":{"1min":10.414,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":31.209,"5min":31.209,"15min":117.10299999999999},"last":13.209},{"interface":"front","average":{"1min":20.265999999999998,"5min":17.077000000000002,"15min":24.907},"min":{"1min":6.9480000000000004,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":32.838999999999999,"5min":32.838999999999999,"15min":111.874},"last":21.677}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":12.294,"5min":11.125,"15min":15.733000000000001},"min":{"1min":7.5800000000000001,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":19.867999999999999,"5min":22.010000000000002,"15min":39.886000000000003},"last":25.277000000000001},{"interface":"front","average":{"1min":19.161999999999999,"5min":16.074000000000002,"15min":21.274000000000001},"min":{"1min":9.3499999999999996,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":28.93,"5min":30.199999999999999,"15min":60.530999999999999},"last":20.940000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":19.617000000000001,"5min":20.466000000000001,"15min":20.608000000000001},"min":{"1min":12.303000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":28.280999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":13.792999999999999},{"interface":"front","average":{"1min":15.759,"5min":15.573,"15min":15.542},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":29.145,"5min":29.145,"15min":29.145},"last":27.050999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":5.5490000000000004},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":10.912000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":6.6600000000000001},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":15.840999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":10.138},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":9.609}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":16.420999999999999},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":12.558}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":13.67},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":14.558999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":15.148999999999999},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":16.992999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":14.670999999999999,"15min":14.456},"min":{"1min":7.3760000000000003,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":52.643999999999998,"5min":52.643999999999998,"15min":52.643999999999998},"last":11.836},{"interface":"front","average":{"1min":18.873999999999999,"5min":18.574999999999999,"15min":18.524999999999999},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":55.177999999999997,"5min":55.177999999999997,"15min":55.177999999999997},"last":18.492999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":7.1870000000000003},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":17.106999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":15.488},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":20.204000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":22.738},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":6.4290000000000003}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":8.2349999999999994},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":24.486000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":25.449999999999999},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":8.8460000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":21.367999999999999},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":18.015000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":18.491,"5min":19.448,"15min":19.608000000000001},"min":{"1min":12.093,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":26.370999999999999,"5min":32.191000000000003,"15min":32.191000000000003},"last":9.5739999999999998},{"interface":"front","average":{"1min":18.702999999999999,"5min":19.315000000000001,"15min":19.417000000000002},"min":{"1min":10.683,"5min":10.683,"15min":10.683},"max":{"1min":35.384999999999998,"5min":36.454000000000001,"15min":36.454000000000001},"last":20.795000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1028096,"data_stored":368992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":6.141},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":10.113}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":8.8810000000000002},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":20.882000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":21.478000000000002},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":9.5}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":8.1270000000000007},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":19.992999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":22.102},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":12.932}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":6.9119999999999999},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":23.234999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":18.797000000000001,"5min":17.757000000000001,"15min":17.582999999999998},"min":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":32.920000000000002,"5min":32.920000000000002,"15min":32.920000000000002},"last":10.731},{"interface":"front","average":{"1min":20.239999999999998,"5min":18.984000000000002,"15min":18.774999999999999},"min":{"1min":11.968,"5min":11.663,"15min":11.663},"max":{"1min":41.348999999999997,"5min":41.348999999999997,"15min":41.348999999999997},"last":23.869}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:28:33.936 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:28:34.532 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:34.532 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:28:34.545 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2023-12-06T09:28:33.987305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225336,"kb_used_data":8184,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749384,"statfs":{"total":773094113280,"available":772863369216,"internally_reserved":0,"allocated":8380416,"data_stored":3778276,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001395"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.988825+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:28:26.988825+0000","last_peered":"2023-12-06T09:28:26.988825+0000","last_clean":"2023-12-06T09:28:26.988825+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:26.988825+0000","last_undegraded":"2023-12-06T09:28:26.988825+0000","last_fullsized":"2023-12-06T09:28:26.988825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.911896+0000","last_change":"2023-12-06T09:28:08.040199+0000","last_active":"2023-12-06T09:28:27.911896+0000","last_peered":"2023-12-06T09:28:27.911896+0000","last_clean":"2023-12-06T09:28:27.911896+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:27.911896+0000","last_undegraded":"2023-12-06T09:28:27.911896+0000","last_fullsized":"2023-12-06T09:28:27.911896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:42:47.793196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.734283+0000","last_change":"2023-12-06T09:28:07.998592+0000","last_active":"2023-12-06T09:28:27.734283+0000","last_peered":"2023-12-06T09:28:27.734283+0000","last_clean":"2023-12-06T09:28:27.734283+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:27.734283+0000","last_undegraded":"2023-12-06T09:28:27.734283+0000","last_fullsized":"2023-12-06T09:28:27.734283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:46:54.460756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.903567+0000","last_change":"2023-12-06T09:28:06.903121+0000","last_active":"2023-12-06T09:28:27.903567+0000","last_peered":"2023-12-06T09:28:27.903567+0000","last_clean":"2023-12-06T09:28:27.903567+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:27.903567+0000","last_undegraded":"2023-12-06T09:28:27.903567+0000","last_fullsized":"2023-12-06T09:28:27.903567+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:49:16.145368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.730818+0000","last_change":"2023-12-06T09:28:18.783833+0000","last_active":"2023-12-06T09:28:27.730818+0000","last_peered":"2023-12-06T09:28:27.730818+0000","last_clean":"2023-12-06T09:28:27.730818+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:27.730818+0000","last_undegraded":"2023-12-06T09:28:27.730818+0000","last_fullsized":"2023-12-06T09:28:27.730818+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:48:37.840608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.983941+0000","last_change":"2023-12-06T09:28:00.969260+0000","last_active":"2023-12-06T09:28:26.983941+0000","last_peered":"2023-12-06T09:28:26.983941+0000","last_clean":"2023-12-06T09:28:26.983941+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:26.983941+0000","last_undegraded":"2023-12-06T09:28:26.983941+0000","last_fullsized":"2023-12-06T09:28:26.983941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:54:26.793621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.736127+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:28:27.736127+0000","last_peered":"2023-12-06T09:28:27.736127+0000","last_clean":"2023-12-06T09:28:27.736127+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:27.736127+0000","last_undegraded":"2023-12-06T09:28:27.736127+0000","last_fullsized":"2023-12-06T09:28:27.736127+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.732589+0000","last_change":"2023-12-06T09:28:18.640184+0000","last_active":"2023-12-06T09:28:27.732589+0000","last_peered":"2023-12-06T09:28:27.732589+0000","last_clean":"2023-12-06T09:28:27.732589+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:27.732589+0000","last_undegraded":"2023-12-06T09:28:27.732589+0000","last_fullsized":"2023-12-06T09:28:27.732589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:06:02.343820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":243,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:28.786089+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:28:28.786089+0000","last_peered":"2023-12-06T09:28:28.786089+0000","last_clean":"2023-12-06T09:28:28.786089+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:28.786089+0000","last_undegraded":"2023-12-06T09:28:28.786089+0000","last_fullsized":"2023-12-06T09:28:28.786089+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":49.530000000000001},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":45.340000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":24.831},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.161000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":26.298999999999999},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":29.259}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":30.797999999999998},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":32.622}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":34.509999999999998},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":27.149000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":27.738},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":13.734999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:58 2023","interfaces":[{"interface":"back","average":{"1min":23.475000000000001,"5min":23.638999999999999,"15min":23.667000000000002},"min":{"1min":11.068,"5min":11.068,"15min":11.068},"max":{"1min":45.051000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":21.917000000000002},{"interface":"front","average":{"1min":22.222000000000001,"5min":22.856999999999999,"15min":22.963000000000001},"min":{"1min":13.550000000000001,"5min":11.01,"15min":11.01},"max":{"1min":41.414000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":21.184000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":7.907},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":15.266999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":6.9169999999999998},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":8.7910000000000004}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":13.186999999999999},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":12.784000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":19.539000000000001},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":11.821999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":20.539000000000001},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":22.658000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":14.348000000000001},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":18.431000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":17.5},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":23.75}]}]},{"osd":0,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":12.863},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":12.300000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":14.683},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":16.667999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":6.7539999999999996},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":16.056000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":8.8699999999999992},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":18.913}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":17.905000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":10.766}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":22.132999999999999},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":20.402999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":20.242999999999999,"5min":19.175000000000001,"15min":18.997},"min":{"1min":12.217000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":44.826000000000001,"5min":61.445999999999998,"15min":61.445999999999998},"last":13.907},{"interface":"front","average":{"1min":22.859000000000002,"5min":20.625,"15min":20.253},"min":{"1min":13.35,"5min":11.833,"15min":11.833},"max":{"1min":38.460000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":21.338999999999999}]}]},{"osd":7,"up_from":60,"seq":257698037794,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":4.0700000000000003},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":8.1699999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":18.030000000000001},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":13.34}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":7.0209999999999999},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":14.106999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":12.182},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":14.749000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":17.204999999999998},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":9.2629999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":10.500999999999999},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":15.430999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":18.806000000000001},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":11.301}]}]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":4.1959999999999997},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":7.343}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.141,"5min":12.663,"15min":14.898999999999999},"min":{"1min":7.5880000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":27.593,"5min":37.103999999999999,"15min":55.398000000000003},"last":15.6},{"interface":"front","average":{"1min":14.585000000000001,"5min":13.566000000000001,"15min":14.862},"min":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"max":{"1min":25.289000000000001,"5min":38.137,"15min":41.811999999999998},"last":12.631}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":8.1500000000000004},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":10.66}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":9.9789999999999992},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":12.015000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":19.532,"5min":15.497,"15min":22.34},"min":{"1min":10.414,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":31.209,"5min":31.209,"15min":117.10299999999999},"last":13.209},{"interface":"front","average":{"1min":20.265999999999998,"5min":17.077000000000002,"15min":24.907},"min":{"1min":6.9480000000000004,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":32.838999999999999,"5min":32.838999999999999,"15min":111.874},"last":21.677}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":12.294,"5min":11.125,"15min":15.733000000000001},"min":{"1min":7.5800000000000001,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":19.867999999999999,"5min":22.010000000000002,"15min":39.886000000000003},"last":25.277000000000001},{"interface":"front","average":{"1min":19.161999999999999,"5min":16.074000000000002,"15min":21.274000000000001},"min":{"1min":9.3499999999999996,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":28.93,"5min":30.199999999999999,"15min":60.530999999999999},"last":20.940000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":19.617000000000001,"5min":20.466000000000001,"15min":20.608000000000001},"min":{"1min":12.303000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":28.280999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":13.792999999999999},{"interface":"front","average":{"1min":15.759,"5min":15.573,"15min":15.542},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":29.145,"5min":29.145,"15min":29.145},"last":27.050999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":5.5490000000000004},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":10.912000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":6.6600000000000001},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":15.840999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":10.138},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":9.609}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":16.420999999999999},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":12.558}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":13.67},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":14.558999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":15.148999999999999},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":16.992999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":14.670999999999999,"15min":14.456},"min":{"1min":7.3760000000000003,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":52.643999999999998,"5min":52.643999999999998,"15min":52.643999999999998},"last":11.836},{"interface":"front","average":{"1min":18.873999999999999,"5min":18.574999999999999,"15min":18.524999999999999},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":55.177999999999997,"5min":55.177999999999997,"15min":55.177999999999997},"last":18.492999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":7.1870000000000003},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":17.106999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":15.488},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":20.204000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":22.738},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":6.4290000000000003}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":8.2349999999999994},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":24.486000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":25.449999999999999},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":8.8460000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":21.367999999999999},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":18.015000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":18.491,"5min":19.448,"15min":19.608000000000001},"min":{"1min":12.093,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":26.370999999999999,"5min":32.191000000000003,"15min":32.191000000000003},"last":9.5739999999999998},{"interface":"front","average":{"1min":18.702999999999999,"5min":19.315000000000001,"15min":19.417000000000002},"min":{"1min":10.683,"5min":10.683,"15min":10.683},"max":{"1min":35.384999999999998,"5min":36.454000000000001,"15min":36.454000000000001},"last":20.795000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1028096,"data_stored":368992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":6.141},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":10.113}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":8.8810000000000002},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":20.882000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":21.478000000000002},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":9.5}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":8.1270000000000007},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":19.992999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":22.102},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":12.932}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":6.9119999999999999},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":23.234999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":18.797000000000001,"5min":17.757000000000001,"15min":17.582999999999998},"min":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":32.920000000000002,"5min":32.920000000000002,"15min":32.920000000000002},"last":10.731},{"interface":"front","average":{"1min":20.239999999999998,"5min":18.984000000000002,"15min":18.774999999999999},"min":{"1min":11.968,"5min":11.663,"15min":11.663},"max":{"1min":41.348999999999997,"5min":41.348999999999997,"15min":41.348999999999997},"last":23.869}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:28:34.546 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-06T09:28:34.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2023-12-06T09:28:35.287 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-06T09:28:35.297 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-06T09:28:35.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:28:35.848 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:35.849 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:28:35.860 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2023-12-06T09:28:33.987305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225336,"kb_used_data":8184,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749384,"statfs":{"total":773094113280,"available":772863369216,"internally_reserved":0,"allocated":8380416,"data_stored":3778276,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001395"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.988825+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:28:26.988825+0000","last_peered":"2023-12-06T09:28:26.988825+0000","last_clean":"2023-12-06T09:28:26.988825+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:26.988825+0000","last_undegraded":"2023-12-06T09:28:26.988825+0000","last_fullsized":"2023-12-06T09:28:26.988825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.911896+0000","last_change":"2023-12-06T09:28:08.040199+0000","last_active":"2023-12-06T09:28:27.911896+0000","last_peered":"2023-12-06T09:28:27.911896+0000","last_clean":"2023-12-06T09:28:27.911896+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:27.911896+0000","last_undegraded":"2023-12-06T09:28:27.911896+0000","last_fullsized":"2023-12-06T09:28:27.911896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:42:47.793196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.734283+0000","last_change":"2023-12-06T09:28:07.998592+0000","last_active":"2023-12-06T09:28:27.734283+0000","last_peered":"2023-12-06T09:28:27.734283+0000","last_clean":"2023-12-06T09:28:27.734283+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:27.734283+0000","last_undegraded":"2023-12-06T09:28:27.734283+0000","last_fullsized":"2023-12-06T09:28:27.734283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:46:54.460756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.903567+0000","last_change":"2023-12-06T09:28:06.903121+0000","last_active":"2023-12-06T09:28:27.903567+0000","last_peered":"2023-12-06T09:28:27.903567+0000","last_clean":"2023-12-06T09:28:27.903567+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:27.903567+0000","last_undegraded":"2023-12-06T09:28:27.903567+0000","last_fullsized":"2023-12-06T09:28:27.903567+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:49:16.145368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.730818+0000","last_change":"2023-12-06T09:28:18.783833+0000","last_active":"2023-12-06T09:28:27.730818+0000","last_peered":"2023-12-06T09:28:27.730818+0000","last_clean":"2023-12-06T09:28:27.730818+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:27.730818+0000","last_undegraded":"2023-12-06T09:28:27.730818+0000","last_fullsized":"2023-12-06T09:28:27.730818+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:48:37.840608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.983941+0000","last_change":"2023-12-06T09:28:00.969260+0000","last_active":"2023-12-06T09:28:26.983941+0000","last_peered":"2023-12-06T09:28:26.983941+0000","last_clean":"2023-12-06T09:28:26.983941+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:26.983941+0000","last_undegraded":"2023-12-06T09:28:26.983941+0000","last_fullsized":"2023-12-06T09:28:26.983941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:54:26.793621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.736127+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:28:27.736127+0000","last_peered":"2023-12-06T09:28:27.736127+0000","last_clean":"2023-12-06T09:28:27.736127+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:27.736127+0000","last_undegraded":"2023-12-06T09:28:27.736127+0000","last_fullsized":"2023-12-06T09:28:27.736127+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.732589+0000","last_change":"2023-12-06T09:28:18.640184+0000","last_active":"2023-12-06T09:28:27.732589+0000","last_peered":"2023-12-06T09:28:27.732589+0000","last_clean":"2023-12-06T09:28:27.732589+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:27.732589+0000","last_undegraded":"2023-12-06T09:28:27.732589+0000","last_fullsized":"2023-12-06T09:28:27.732589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:06:02.343820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":243,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:28.786089+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:28:28.786089+0000","last_peered":"2023-12-06T09:28:28.786089+0000","last_clean":"2023-12-06T09:28:28.786089+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:28.786089+0000","last_undegraded":"2023-12-06T09:28:28.786089+0000","last_fullsized":"2023-12-06T09:28:28.786089+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":49.530000000000001},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":45.340000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":24.831},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.161000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":26.298999999999999},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":29.259}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":30.797999999999998},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":32.622}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":34.509999999999998},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":27.149000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":27.738},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":13.734999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:58 2023","interfaces":[{"interface":"back","average":{"1min":23.475000000000001,"5min":23.638999999999999,"15min":23.667000000000002},"min":{"1min":11.068,"5min":11.068,"15min":11.068},"max":{"1min":45.051000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":21.917000000000002},{"interface":"front","average":{"1min":22.222000000000001,"5min":22.856999999999999,"15min":22.963000000000001},"min":{"1min":13.550000000000001,"5min":11.01,"15min":11.01},"max":{"1min":41.414000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":21.184000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":7.907},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":15.266999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":6.9169999999999998},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":8.7910000000000004}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":13.186999999999999},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":12.784000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":19.539000000000001},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":11.821999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":20.539000000000001},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":22.658000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":14.348000000000001},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":18.431000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":17.5},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":23.75}]}]},{"osd":0,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":12.863},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":12.300000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":14.683},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":16.667999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":6.7539999999999996},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":16.056000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":8.8699999999999992},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":18.913}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":17.905000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":10.766}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":22.132999999999999},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":20.402999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":20.242999999999999,"5min":19.175000000000001,"15min":18.997},"min":{"1min":12.217000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":44.826000000000001,"5min":61.445999999999998,"15min":61.445999999999998},"last":13.907},{"interface":"front","average":{"1min":22.859000000000002,"5min":20.625,"15min":20.253},"min":{"1min":13.35,"5min":11.833,"15min":11.833},"max":{"1min":38.460000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":21.338999999999999}]}]},{"osd":7,"up_from":60,"seq":257698037794,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":4.0700000000000003},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":8.1699999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":18.030000000000001},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":13.34}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":7.0209999999999999},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":14.106999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":12.182},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":14.749000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":17.204999999999998},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":9.2629999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":10.500999999999999},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":15.430999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":18.806000000000001},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":11.301}]}]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":4.1959999999999997},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":7.343}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.141,"5min":12.663,"15min":14.898999999999999},"min":{"1min":7.5880000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":27.593,"5min":37.103999999999999,"15min":55.398000000000003},"last":15.6},{"interface":"front","average":{"1min":14.585000000000001,"5min":13.566000000000001,"15min":14.862},"min":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"max":{"1min":25.289000000000001,"5min":38.137,"15min":41.811999999999998},"last":12.631}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":8.1500000000000004},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":10.66}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":9.9789999999999992},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":12.015000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":19.532,"5min":15.497,"15min":22.34},"min":{"1min":10.414,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":31.209,"5min":31.209,"15min":117.10299999999999},"last":13.209},{"interface":"front","average":{"1min":20.265999999999998,"5min":17.077000000000002,"15min":24.907},"min":{"1min":6.9480000000000004,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":32.838999999999999,"5min":32.838999999999999,"15min":111.874},"last":21.677}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":12.294,"5min":11.125,"15min":15.733000000000001},"min":{"1min":7.5800000000000001,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":19.867999999999999,"5min":22.010000000000002,"15min":39.886000000000003},"last":25.277000000000001},{"interface":"front","average":{"1min":19.161999999999999,"5min":16.074000000000002,"15min":21.274000000000001},"min":{"1min":9.3499999999999996,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":28.93,"5min":30.199999999999999,"15min":60.530999999999999},"last":20.940000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":19.617000000000001,"5min":20.466000000000001,"15min":20.608000000000001},"min":{"1min":12.303000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":28.280999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":13.792999999999999},{"interface":"front","average":{"1min":15.759,"5min":15.573,"15min":15.542},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":29.145,"5min":29.145,"15min":29.145},"last":27.050999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":5.5490000000000004},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":10.912000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":6.6600000000000001},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":15.840999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":10.138},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":9.609}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":16.420999999999999},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":12.558}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":13.67},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":14.558999999999999}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":15.148999999999999},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":16.992999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":14.670999999999999,"15min":14.456},"min":{"1min":7.3760000000000003,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":52.643999999999998,"5min":52.643999999999998,"15min":52.643999999999998},"last":11.836},{"interface":"front","average":{"1min":18.873999999999999,"5min":18.574999999999999,"15min":18.524999999999999},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":55.177999999999997,"5min":55.177999999999997,"15min":55.177999999999997},"last":18.492999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":7.1870000000000003},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":17.106999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":15.488},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":20.204000000000001}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":22.738},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":6.4290000000000003}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":8.2349999999999994},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":24.486000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":25.449999999999999},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":8.8460000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":21.367999999999999},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":18.015000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":18.491,"5min":19.448,"15min":19.608000000000001},"min":{"1min":12.093,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":26.370999999999999,"5min":32.191000000000003,"15min":32.191000000000003},"last":9.5739999999999998},{"interface":"front","average":{"1min":18.702999999999999,"5min":19.315000000000001,"15min":19.417000000000002},"min":{"1min":10.683,"5min":10.683,"15min":10.683},"max":{"1min":35.384999999999998,"5min":36.454000000000001,"15min":36.454000000000001},"last":20.795000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1028096,"data_stored":368992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":6.141},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":10.113}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":8.8810000000000002},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":20.882000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":21.478000000000002},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":9.5}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":8.1270000000000007},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":19.992999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":22.102},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":12.932}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":6.9119999999999999},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":23.234999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":18.797000000000001,"5min":17.757000000000001,"15min":17.582999999999998},"min":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":32.920000000000002,"5min":32.920000000000002,"15min":32.920000000000002},"last":10.731},{"interface":"front","average":{"1min":20.239999999999998,"5min":18.984000000000002,"15min":18.774999999999999},"min":{"1min":11.968,"5min":11.663,"15min":11.663},"max":{"1min":41.348999999999997,"5min":41.348999999999997,"15min":41.348999999999997},"last":23.869}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:28:35.862 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:28:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:36.418 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:28:36.430 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2023-12-06T09:28:35.987579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225352,"kb_used_data":8196,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749368,"statfs":{"total":773094113280,"available":772863352832,"internally_reserved":0,"allocated":8392704,"data_stored":3782656,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.053817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.988825+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:28:26.988825+0000","last_peered":"2023-12-06T09:28:26.988825+0000","last_clean":"2023-12-06T09:28:26.988825+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:26.988825+0000","last_undegraded":"2023-12-06T09:28:26.988825+0000","last_fullsized":"2023-12-06T09:28:26.988825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.911896+0000","last_change":"2023-12-06T09:28:08.040199+0000","last_active":"2023-12-06T09:28:27.911896+0000","last_peered":"2023-12-06T09:28:27.911896+0000","last_clean":"2023-12-06T09:28:27.911896+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:27.911896+0000","last_undegraded":"2023-12-06T09:28:27.911896+0000","last_fullsized":"2023-12-06T09:28:27.911896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:42:47.793196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.734283+0000","last_change":"2023-12-06T09:28:07.998592+0000","last_active":"2023-12-06T09:28:27.734283+0000","last_peered":"2023-12-06T09:28:27.734283+0000","last_clean":"2023-12-06T09:28:27.734283+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:27.734283+0000","last_undegraded":"2023-12-06T09:28:27.734283+0000","last_fullsized":"2023-12-06T09:28:27.734283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:46:54.460756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.903567+0000","last_change":"2023-12-06T09:28:06.903121+0000","last_active":"2023-12-06T09:28:27.903567+0000","last_peered":"2023-12-06T09:28:27.903567+0000","last_clean":"2023-12-06T09:28:27.903567+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:27.903567+0000","last_undegraded":"2023-12-06T09:28:27.903567+0000","last_fullsized":"2023-12-06T09:28:27.903567+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:49:16.145368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.730818+0000","last_change":"2023-12-06T09:28:18.783833+0000","last_active":"2023-12-06T09:28:27.730818+0000","last_peered":"2023-12-06T09:28:27.730818+0000","last_clean":"2023-12-06T09:28:27.730818+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:27.730818+0000","last_undegraded":"2023-12-06T09:28:27.730818+0000","last_fullsized":"2023-12-06T09:28:27.730818+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:48:37.840608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.983941+0000","last_change":"2023-12-06T09:28:00.969260+0000","last_active":"2023-12-06T09:28:26.983941+0000","last_peered":"2023-12-06T09:28:26.983941+0000","last_clean":"2023-12-06T09:28:26.983941+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:26.983941+0000","last_undegraded":"2023-12-06T09:28:26.983941+0000","last_fullsized":"2023-12-06T09:28:26.983941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:54:26.793621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.736127+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:28:27.736127+0000","last_peered":"2023-12-06T09:28:27.736127+0000","last_clean":"2023-12-06T09:28:27.736127+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:27.736127+0000","last_undegraded":"2023-12-06T09:28:27.736127+0000","last_fullsized":"2023-12-06T09:28:27.736127+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.732589+0000","last_change":"2023-12-06T09:28:18.640184+0000","last_active":"2023-12-06T09:28:27.732589+0000","last_peered":"2023-12-06T09:28:27.732589+0000","last_clean":"2023-12-06T09:28:27.732589+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:27.732589+0000","last_undegraded":"2023-12-06T09:28:27.732589+0000","last_fullsized":"2023-12-06T09:28:27.732589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:06:02.343820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":243,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:28.786089+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:28:28.786089+0000","last_peered":"2023-12-06T09:28:28.786089+0000","last_clean":"2023-12-06T09:28:28.786089+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:28.786089+0000","last_undegraded":"2023-12-06T09:28:28.786089+0000","last_fullsized":"2023-12-06T09:28:28.786089+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":49.530000000000001},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":45.340000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":24.831},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.161000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":26.298999999999999},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":29.259}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":30.797999999999998},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":32.622}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":34.509999999999998},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":27.149000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":27.738},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":13.734999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:58 2023","interfaces":[{"interface":"back","average":{"1min":23.475000000000001,"5min":23.638999999999999,"15min":23.667000000000002},"min":{"1min":11.068,"5min":11.068,"15min":11.068},"max":{"1min":45.051000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":21.917000000000002},{"interface":"front","average":{"1min":22.222000000000001,"5min":22.856999999999999,"15min":22.963000000000001},"min":{"1min":13.550000000000001,"5min":11.01,"15min":11.01},"max":{"1min":41.414000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":21.184000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":7.907},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":15.266999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":6.9169999999999998},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":8.7910000000000004}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":13.186999999999999},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":12.784000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":19.539000000000001},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":11.821999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":20.539000000000001},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":22.658000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":14.348000000000001},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":18.431000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":17.5},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":23.75}]}]},{"osd":0,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":12.863},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":12.300000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":14.683},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":16.667999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":6.7539999999999996},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":16.056000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":8.8699999999999992},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":18.913}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":17.905000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":10.766}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":22.132999999999999},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":20.402999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":20.242999999999999,"5min":19.175000000000001,"15min":18.997},"min":{"1min":12.217000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":44.826000000000001,"5min":61.445999999999998,"15min":61.445999999999998},"last":13.907},{"interface":"front","average":{"1min":22.859000000000002,"5min":20.625,"15min":20.253},"min":{"1min":13.35,"5min":11.833,"15min":11.833},"max":{"1min":38.460000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":21.338999999999999}]}]},{"osd":7,"up_from":60,"seq":257698037795,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":4.0099999999999998},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":8.3499999999999996}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":11.648999999999999},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":15.090999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":8.7539999999999996},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":14.478}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":12.815},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":13.470000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":17.870000000000001},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":7.9370000000000003}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":10.976000000000001},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":16.239000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":17.120999999999999},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":10.374000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":8.7289999999999992},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":8.1760000000000002}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.141,"5min":12.663,"15min":14.898999999999999},"min":{"1min":7.5880000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":27.593,"5min":37.103999999999999,"15min":55.398000000000003},"last":10.579000000000001},{"interface":"front","average":{"1min":14.585000000000001,"5min":13.566000000000001,"15min":14.862},"min":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"max":{"1min":25.289000000000001,"5min":38.137,"15min":41.811999999999998},"last":7.8520000000000003}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":13.923999999999999},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":9.891}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":9.3309999999999995},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":11.843}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":19.532,"5min":15.497,"15min":22.34},"min":{"1min":10.414,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":31.209,"5min":31.209,"15min":117.10299999999999},"last":15.141},{"interface":"front","average":{"1min":20.265999999999998,"5min":17.077000000000002,"15min":24.907},"min":{"1min":6.9480000000000004,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":32.838999999999999,"5min":32.838999999999999,"15min":111.874},"last":18.456}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":12.294,"5min":11.125,"15min":15.733000000000001},"min":{"1min":7.5800000000000001,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":19.867999999999999,"5min":22.010000000000002,"15min":39.886000000000003},"last":16.504999999999999},{"interface":"front","average":{"1min":19.161999999999999,"5min":16.074000000000002,"15min":21.274000000000001},"min":{"1min":9.3499999999999996,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":28.93,"5min":30.199999999999999,"15min":60.530999999999999},"last":12.646000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":19.617000000000001,"5min":20.466000000000001,"15min":20.608000000000001},"min":{"1min":12.303000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":28.280999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":14.555999999999999},{"interface":"front","average":{"1min":15.759,"5min":15.573,"15min":15.542},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":29.145,"5min":29.145,"15min":29.145},"last":17.495999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542227,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":15.736000000000001},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":11.539}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":19.366},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":13.257999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":7.532},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":16.413}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":9.5399999999999991},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":13.939}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":9.2360000000000007},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":20.271000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":18.013000000000002},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":12.148}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":14.670999999999999,"15min":14.456},"min":{"1min":7.3760000000000003,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":52.643999999999998,"5min":52.643999999999998,"15min":52.643999999999998},"last":17.088999999999999},{"interface":"front","average":{"1min":18.873999999999999,"5min":18.574999999999999,"15min":18.524999999999999},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":55.177999999999997,"5min":55.177999999999997,"15min":55.177999999999997},"last":12.683}]}]},{"osd":1,"up_from":14,"seq":60129542227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":5.3810000000000002},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":10.061}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":11.321999999999999},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":16.265999999999998}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":10.994999999999999},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":7.3559999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":10.404},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":12.146000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":25.001000000000001},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":21.922000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":17.234000000000002},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":14.130000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":18.491,"5min":19.448,"15min":19.608000000000001},"min":{"1min":12.093,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":26.370999999999999,"5min":32.191000000000003,"15min":32.191000000000003},"last":18.457000000000001},{"interface":"front","average":{"1min":18.702999999999999,"5min":19.315000000000001,"15min":19.417000000000002},"min":{"1min":10.683,"5min":10.683,"15min":10.683},"max":{"1min":35.384999999999998,"5min":36.454000000000001,"15min":36.454000000000001},"last":19.658999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":5.1820000000000004},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":6.8760000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":14.076000000000001},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":8.3399999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":9.2710000000000008},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":17.465}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":9.9589999999999996},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":16.494}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":11.696},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":13.484}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":18.667000000000002},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":12.787000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":18.797000000000001,"5min":17.757000000000001,"15min":17.582999999999998},"min":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":32.920000000000002,"5min":32.920000000000002,"15min":32.920000000000002},"last":19.635000000000002},{"interface":"front","average":{"1min":20.239999999999998,"5min":18.984000000000002,"15min":18.774999999999999},"min":{"1min":11.968,"5min":11.663,"15min":11.663},"max":{"1min":41.348999999999997,"5min":41.348999999999997,"15min":41.348999999999997},"last":15.541}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:28:36.432 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-06T09:28:36.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:28:37.008 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:37.008 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:28:37.020 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2023-12-06T09:28:35.987579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225352,"kb_used_data":8196,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749368,"statfs":{"total":773094113280,"available":772863352832,"internally_reserved":0,"allocated":8392704,"data_stored":3782656,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.053817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.988825+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:28:26.988825+0000","last_peered":"2023-12-06T09:28:26.988825+0000","last_clean":"2023-12-06T09:28:26.988825+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:26.988825+0000","last_undegraded":"2023-12-06T09:28:26.988825+0000","last_fullsized":"2023-12-06T09:28:26.988825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.911896+0000","last_change":"2023-12-06T09:28:08.040199+0000","last_active":"2023-12-06T09:28:27.911896+0000","last_peered":"2023-12-06T09:28:27.911896+0000","last_clean":"2023-12-06T09:28:27.911896+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:27.911896+0000","last_undegraded":"2023-12-06T09:28:27.911896+0000","last_fullsized":"2023-12-06T09:28:27.911896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:42:47.793196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.734283+0000","last_change":"2023-12-06T09:28:07.998592+0000","last_active":"2023-12-06T09:28:27.734283+0000","last_peered":"2023-12-06T09:28:27.734283+0000","last_clean":"2023-12-06T09:28:27.734283+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:27.734283+0000","last_undegraded":"2023-12-06T09:28:27.734283+0000","last_fullsized":"2023-12-06T09:28:27.734283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:46:54.460756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.903567+0000","last_change":"2023-12-06T09:28:06.903121+0000","last_active":"2023-12-06T09:28:27.903567+0000","last_peered":"2023-12-06T09:28:27.903567+0000","last_clean":"2023-12-06T09:28:27.903567+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:27.903567+0000","last_undegraded":"2023-12-06T09:28:27.903567+0000","last_fullsized":"2023-12-06T09:28:27.903567+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:49:16.145368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.730818+0000","last_change":"2023-12-06T09:28:18.783833+0000","last_active":"2023-12-06T09:28:27.730818+0000","last_peered":"2023-12-06T09:28:27.730818+0000","last_clean":"2023-12-06T09:28:27.730818+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:27.730818+0000","last_undegraded":"2023-12-06T09:28:27.730818+0000","last_fullsized":"2023-12-06T09:28:27.730818+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:48:37.840608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.983941+0000","last_change":"2023-12-06T09:28:00.969260+0000","last_active":"2023-12-06T09:28:26.983941+0000","last_peered":"2023-12-06T09:28:26.983941+0000","last_clean":"2023-12-06T09:28:26.983941+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:26.983941+0000","last_undegraded":"2023-12-06T09:28:26.983941+0000","last_fullsized":"2023-12-06T09:28:26.983941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:54:26.793621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.736127+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:28:27.736127+0000","last_peered":"2023-12-06T09:28:27.736127+0000","last_clean":"2023-12-06T09:28:27.736127+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:27.736127+0000","last_undegraded":"2023-12-06T09:28:27.736127+0000","last_fullsized":"2023-12-06T09:28:27.736127+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.732589+0000","last_change":"2023-12-06T09:28:18.640184+0000","last_active":"2023-12-06T09:28:27.732589+0000","last_peered":"2023-12-06T09:28:27.732589+0000","last_clean":"2023-12-06T09:28:27.732589+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:27.732589+0000","last_undegraded":"2023-12-06T09:28:27.732589+0000","last_fullsized":"2023-12-06T09:28:27.732589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:06:02.343820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":243,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:28.786089+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:28:28.786089+0000","last_peered":"2023-12-06T09:28:28.786089+0000","last_clean":"2023-12-06T09:28:28.786089+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:28.786089+0000","last_undegraded":"2023-12-06T09:28:28.786089+0000","last_fullsized":"2023-12-06T09:28:28.786089+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":49.530000000000001},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":45.340000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":24.831},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":24.161000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":26.298999999999999},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":29.259}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":30.797999999999998},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":32.622}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":34.509999999999998},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":27.149000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":27.738},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":13.734999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:58 2023","interfaces":[{"interface":"back","average":{"1min":23.475000000000001,"5min":23.638999999999999,"15min":23.667000000000002},"min":{"1min":11.068,"5min":11.068,"15min":11.068},"max":{"1min":45.051000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":21.917000000000002},{"interface":"front","average":{"1min":22.222000000000001,"5min":22.856999999999999,"15min":22.963000000000001},"min":{"1min":13.550000000000001,"5min":11.01,"15min":11.01},"max":{"1min":41.414000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":21.184000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":7.907},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":15.266999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":6.9169999999999998},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":8.7910000000000004}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":13.186999999999999},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":12.784000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":19.539000000000001},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":11.821999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":20.539000000000001},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":22.658000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":14.348000000000001},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":18.431000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":17.5},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":23.75}]}]},{"osd":0,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":12.863},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":12.300000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":14.683},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":16.667999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":6.7539999999999996},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":16.056000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":8.8699999999999992},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":18.913}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":17.905000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":10.766}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":22.132999999999999},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":20.402999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":20.242999999999999,"5min":19.175000000000001,"15min":18.997},"min":{"1min":12.217000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":44.826000000000001,"5min":61.445999999999998,"15min":61.445999999999998},"last":13.907},{"interface":"front","average":{"1min":22.859000000000002,"5min":20.625,"15min":20.253},"min":{"1min":13.35,"5min":11.833,"15min":11.833},"max":{"1min":38.460000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":21.338999999999999}]}]},{"osd":7,"up_from":60,"seq":257698037795,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":4.0099999999999998},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":8.3499999999999996}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":11.648999999999999},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":15.090999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":8.7539999999999996},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":14.478}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":12.815},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":13.470000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":17.870000000000001},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":7.9370000000000003}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":10.976000000000001},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":16.239000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":17.120999999999999},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":10.374000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":8.7289999999999992},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":8.1760000000000002}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.141,"5min":12.663,"15min":14.898999999999999},"min":{"1min":7.5880000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":27.593,"5min":37.103999999999999,"15min":55.398000000000003},"last":10.579000000000001},{"interface":"front","average":{"1min":14.585000000000001,"5min":13.566000000000001,"15min":14.862},"min":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"max":{"1min":25.289000000000001,"5min":38.137,"15min":41.811999999999998},"last":7.8520000000000003}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":13.923999999999999},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":9.891}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":9.3309999999999995},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":11.843}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":19.532,"5min":15.497,"15min":22.34},"min":{"1min":10.414,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":31.209,"5min":31.209,"15min":117.10299999999999},"last":15.141},{"interface":"front","average":{"1min":20.265999999999998,"5min":17.077000000000002,"15min":24.907},"min":{"1min":6.9480000000000004,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":32.838999999999999,"5min":32.838999999999999,"15min":111.874},"last":18.456}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":12.294,"5min":11.125,"15min":15.733000000000001},"min":{"1min":7.5800000000000001,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":19.867999999999999,"5min":22.010000000000002,"15min":39.886000000000003},"last":16.504999999999999},{"interface":"front","average":{"1min":19.161999999999999,"5min":16.074000000000002,"15min":21.274000000000001},"min":{"1min":9.3499999999999996,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":28.93,"5min":30.199999999999999,"15min":60.530999999999999},"last":12.646000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":19.617000000000001,"5min":20.466000000000001,"15min":20.608000000000001},"min":{"1min":12.303000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":28.280999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":14.555999999999999},{"interface":"front","average":{"1min":15.759,"5min":15.573,"15min":15.542},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":29.145,"5min":29.145,"15min":29.145},"last":17.495999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542227,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":15.736000000000001},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":11.539}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":19.366},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":13.257999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":7.532},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":16.413}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":9.5399999999999991},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":13.939}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":9.2360000000000007},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":20.271000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":18.013000000000002},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":12.148}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":14.670999999999999,"15min":14.456},"min":{"1min":7.3760000000000003,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":52.643999999999998,"5min":52.643999999999998,"15min":52.643999999999998},"last":17.088999999999999},{"interface":"front","average":{"1min":18.873999999999999,"5min":18.574999999999999,"15min":18.524999999999999},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":55.177999999999997,"5min":55.177999999999997,"15min":55.177999999999997},"last":12.683}]}]},{"osd":1,"up_from":14,"seq":60129542227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":5.3810000000000002},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":10.061}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":11.321999999999999},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":16.265999999999998}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":10.994999999999999},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":7.3559999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":10.404},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":12.146000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":25.001000000000001},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":21.922000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":17.234000000000002},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":14.130000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":18.491,"5min":19.448,"15min":19.608000000000001},"min":{"1min":12.093,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":26.370999999999999,"5min":32.191000000000003,"15min":32.191000000000003},"last":18.457000000000001},{"interface":"front","average":{"1min":18.702999999999999,"5min":19.315000000000001,"15min":19.417000000000002},"min":{"1min":10.683,"5min":10.683,"15min":10.683},"max":{"1min":35.384999999999998,"5min":36.454000000000001,"15min":36.454000000000001},"last":19.658999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":5.1820000000000004},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":6.8760000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":14.076000000000001},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":8.3399999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":9.2710000000000008},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":17.465}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":9.9589999999999996},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":16.494}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":11.696},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":13.484}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":18.667000000000002},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":12.787000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":18.797000000000001,"5min":17.757000000000001,"15min":17.582999999999998},"min":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":32.920000000000002,"5min":32.920000000000002,"15min":32.920000000000002},"last":19.635000000000002},{"interface":"front","average":{"1min":20.239999999999998,"5min":18.984000000000002,"15min":18.774999999999999},"min":{"1min":11.968,"5min":11.663,"15min":11.663},"max":{"1min":41.348999999999997,"5min":41.348999999999997,"15min":41.348999999999997},"last":15.541}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:28:37.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T09:28:37.701 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"fsid":"0c48cca3-dcc0-44bd-8eed-37fea2138550","created":"2023-12-06T09:21:08.071156+0000","modified":"2023-12-06T09:28:28.259885+0000","last_up_change":"2023-12-06T09:25:55.041559+0000","last_in_change":"2023-12-06T09:28:06.637886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T09:21:54.386974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T09:22:01.832571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":3.5,"raw_score_stable":3.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db81b709-a318-4b43-beb2-bf3687f365e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2388339328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":2388339328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2388339328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":2388339328}]},"public_addr":"172.21.15.88:6805/2388339328","cluster_addr":"172.21.15.88:6806/2388339328","heartbeat_back_addr":"172.21.15.88:6808/2388339328","heartbeat_front_addr":"172.21.15.88:6807/2388339328","state":["exists","up"]},{"osd":1,"uuid":"fd7b0482-e237-424b-8eff-8db905713049","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":3646708602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3646708602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3646708602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":3646708602}]},"public_addr":"172.21.15.88:6801/3646708602","cluster_addr":"172.21.15.88:6802/3646708602","heartbeat_back_addr":"172.21.15.88:6804/3646708602","heartbeat_front_addr":"172.21.15.88:6803/3646708602","state":["exists","up"]},{"osd":2,"uuid":"e5a571be-6b8e-4c6f-bd3e-2af30c112285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3714168299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3714168299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3714168299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3714168299}]},"public_addr":"172.21.15.88:6810/3714168299","cluster_addr":"172.21.15.88:6813/3714168299","heartbeat_back_addr":"172.21.15.88:6816/3714168299","heartbeat_front_addr":"172.21.15.88:6814/3714168299","state":["exists","up"]},{"osd":3,"uuid":"d7abe19a-cc3d-411f-8533-8592e1379735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":1828251745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":1828251745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6815","nonce":1828251745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1828251745}]},"public_addr":"172.21.15.88:6809/1828251745","cluster_addr":"172.21.15.88:6811/1828251745","heartbeat_back_addr":"172.21.15.88:6815/1828251745","heartbeat_front_addr":"172.21.15.88:6812/1828251745","state":["exists","up"]},{"osd":4,"uuid":"b9517160-d081-44f9-aea4-6d2529e02114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3720978101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6801","nonce":3720978101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3720978101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3720978101}]},"public_addr":"172.21.15.132:6800/3720978101","cluster_addr":"172.21.15.132:6801/3720978101","heartbeat_back_addr":"172.21.15.132:6803/3720978101","heartbeat_front_addr":"172.21.15.132:6802/3720978101","state":["exists","up"]},{"osd":5,"uuid":"b2f9598d-5d7b-4f8b-9fe9-a25c528c2e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1203998358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6805","nonce":1203998358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":1203998358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1203998358}]},"public_addr":"172.21.15.132:6804/1203998358","cluster_addr":"172.21.15.132:6805/1203998358","heartbeat_back_addr":"172.21.15.132:6807/1203998358","heartbeat_front_addr":"172.21.15.132:6806/1203998358","state":["exists","up"]},{"osd":6,"uuid":"e7afeca8-1363-4e9b-96f6-7fab6113d9e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4151455937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6809","nonce":4151455937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":4151455937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4151455937}]},"public_addr":"172.21.15.132:6808/4151455937","cluster_addr":"172.21.15.132:6809/4151455937","heartbeat_back_addr":"172.21.15.132:6811/4151455937","heartbeat_front_addr":"172.21.15.132:6810/4151455937","state":["exists","up"]},{"osd":7,"uuid":"fd4b3971-3adb-44fd-b42b-b6660305beb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":51,"up_from":60,"up_thru":104,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1573021680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":1573021680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6815","nonce":1573021680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1573021680}]},"public_addr":"172.21.15.132:6812/1573021680","cluster_addr":"172.21.15.132:6813/1573021680","heartbeat_back_addr":"172.21.15.132:6815/1573021680","heartbeat_front_addr":"172.21.15.132:6814/1573021680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.229602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.158659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.969728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:53.234010+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:50.601730+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.304649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:51.671399+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T09:21:52.055036+0000","dead_epoch":51}],"pg_upmap":[{"pgid":"2.4","osds":[2,5]},{"pgid":"2.7","osds":[7,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T09:28:38.717 INFO:tasks.ceph:Scrubbing osd.0 2023-12-06T09:28:38.717 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:38.917 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:38.917 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:38.917 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:38.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-06T09:28:39.500 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 0 to deep-scrub 2023-12-06T09:28:39.516 INFO:tasks.ceph:Scrubbing osd.1 2023-12-06T09:28:39.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:39.714 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:39.715 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:39.717 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:39.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-06T09:28:40.292 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 1 to deep-scrub 2023-12-06T09:28:40.309 INFO:tasks.ceph:Scrubbing osd.2 2023-12-06T09:28:40.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:40.519 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:40.519 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:40.519 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:40.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-06T09:28:41.091 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 2 to deep-scrub 2023-12-06T09:28:41.107 INFO:tasks.ceph:Scrubbing osd.3 2023-12-06T09:28:41.107 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:41.304 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:41.304 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:41.304 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:41.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-06T09:28:41.929 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 3 to deep-scrub 2023-12-06T09:28:41.943 INFO:tasks.ceph:Scrubbing osd.4 2023-12-06T09:28:41.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:42.147 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:42.147 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:42.147 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:42.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-06T09:28:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 4 to deep-scrub 2023-12-06T09:28:42.728 INFO:tasks.ceph:Scrubbing osd.5 2023-12-06T09:28:42.728 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:42.976 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:42.976 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:42.976 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:42.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-06T09:28:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 5 to deep-scrub 2023-12-06T09:28:43.597 INFO:tasks.ceph:Scrubbing osd.6 2023-12-06T09:28:43.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:43.787 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:43.787 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:43.787 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:43.797 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-06T09:28:44.355 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 6 to deep-scrub 2023-12-06T09:28:44.369 INFO:tasks.ceph:Scrubbing osd.7 2023-12-06T09:28:44.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-06T09:28:44.582 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-12-06T09:28:44.583 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T09:28:44.583 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-12-06T09:28:44.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-06T09:28:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 7 to deep-scrub 2023-12-06T09:28:45.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:28:45.724 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:28:45.724 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:28:45.736 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2023-12-06T09:28:43.989126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225352,"kb_used_data":8196,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749368,"statfs":{"total":773094113280,"available":772863352832,"internally_reserved":0,"allocated":8392704,"data_stored":3782656,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002210"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.988825+0000","last_change":"2023-12-06T09:25:56.880855+0000","last_active":"2023-12-06T09:28:26.988825+0000","last_peered":"2023-12-06T09:28:26.988825+0000","last_clean":"2023-12-06T09:28:26.988825+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:26.988825+0000","last_undegraded":"2023-12-06T09:28:26.988825+0000","last_fullsized":"2023-12-06T09:28:26.988825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T20:08:23.394931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.911896+0000","last_change":"2023-12-06T09:28:08.040199+0000","last_active":"2023-12-06T09:28:27.911896+0000","last_peered":"2023-12-06T09:28:27.911896+0000","last_clean":"2023-12-06T09:28:27.911896+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:27.911896+0000","last_undegraded":"2023-12-06T09:28:27.911896+0000","last_fullsized":"2023-12-06T09:28:27.911896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:42:47.793196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.734283+0000","last_change":"2023-12-06T09:28:07.998592+0000","last_active":"2023-12-06T09:28:27.734283+0000","last_peered":"2023-12-06T09:28:27.734283+0000","last_clean":"2023-12-06T09:28:27.734283+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:27.734283+0000","last_undegraded":"2023-12-06T09:28:27.734283+0000","last_fullsized":"2023-12-06T09:28:27.734283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:46:54.460756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:41.882791+0000","last_change":"2023-12-06T09:28:41.889541+0000","last_active":"2023-12-06T09:28:41.882791+0000","last_peered":"2023-12-06T09:28:41.882791+0000","last_clean":"2023-12-06T09:28:41.882791+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:41.882791+0000","last_undegraded":"2023-12-06T09:28:41.882791+0000","last_fullsized":"2023-12-06T09:28:41.882791+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:41.878810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:41.878810+0000","last_clean_scrub_stamp":"2023-12-06T09:28:41.878810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:27:42.735018+0000","scrub_duration":0.30635094800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.730818+0000","last_change":"2023-12-06T09:28:18.783833+0000","last_active":"2023-12-06T09:28:27.730818+0000","last_peered":"2023-12-06T09:28:27.730818+0000","last_clean":"2023-12-06T09:28:27.730818+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:27.730818+0000","last_undegraded":"2023-12-06T09:28:27.730818+0000","last_fullsized":"2023-12-06T09:28:27.730818+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:48:37.840608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:26.983941+0000","last_change":"2023-12-06T09:28:00.969260+0000","last_active":"2023-12-06T09:28:26.983941+0000","last_peered":"2023-12-06T09:28:26.983941+0000","last_clean":"2023-12-06T09:28:26.983941+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:26.983941+0000","last_undegraded":"2023-12-06T09:28:26.983941+0000","last_fullsized":"2023-12-06T09:28:26.983941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:54:26.793621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.736127+0000","last_change":"2023-12-06T09:24:37.203532+0000","last_active":"2023-12-06T09:28:27.736127+0000","last_peered":"2023-12-06T09:28:27.736127+0000","last_clean":"2023-12-06T09:28:27.736127+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:27.736127+0000","last_undegraded":"2023-12-06T09:28:27.736127+0000","last_fullsized":"2023-12-06T09:28:27.736127+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:54:19.186499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:27.732589+0000","last_change":"2023-12-06T09:28:18.640184+0000","last_active":"2023-12-06T09:28:27.732589+0000","last_peered":"2023-12-06T09:28:27.732589+0000","last_clean":"2023-12-06T09:28:27.732589+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:27.732589+0000","last_undegraded":"2023-12-06T09:28:27.732589+0000","last_fullsized":"2023-12-06T09:28:27.732589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:22:02.068230+0000","last_clean_scrub_stamp":"2023-12-06T09:22:02.068230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:06:02.343820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":243,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T09:28:28.786089+0000","last_change":"2023-12-06T09:24:37.498869+0000","last_active":"2023-12-06T09:28:28.786089+0000","last_peered":"2023-12-06T09:28:28.786089+0000","last_clean":"2023-12-06T09:28:28.786089+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:28.786089+0000","last_undegraded":"2023-12-06T09:28:28.786089+0000","last_fullsized":"2023-12-06T09:28:28.786089+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:21:55.261571+0000","last_clean_scrub_stamp":"2023-12-06T09:21:55.261571+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:01:21.392951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":7.7969999999999997},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":7.1029999999999998}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":8.8070000000000004},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":15.91}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":10.603},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":11.223000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":9.3849999999999998},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":11.843}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":14.512},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":17.204999999999998}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":15.194000000000001},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":19.042999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:58 2023","interfaces":[{"interface":"back","average":{"1min":23.475000000000001,"5min":23.638999999999999,"15min":23.667000000000002},"min":{"1min":11.068,"5min":11.068,"15min":11.068},"max":{"1min":45.051000000000002,"5min":52.109000000000002,"15min":52.109000000000002},"last":12.553000000000001},{"interface":"front","average":{"1min":22.222000000000001,"5min":22.856999999999999,"15min":22.963000000000001},"min":{"1min":13.550000000000001,"5min":11.01,"15min":11.01},"max":{"1min":41.414000000000001,"5min":56.914000000000001,"15min":56.914000000000001},"last":19.847000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":9.3729999999999993},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":8.7279999999999998}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":11.628},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":13.772}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":11.099},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":14.611000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":16.138000000000002},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":12.263999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":21.341999999999999},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":16.812000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":15.446},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":20.198}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":18.373000000000001},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":18.971}]}]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":7.9550000000000001},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":8.9260000000000002}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":12.077999999999999},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":10.904999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":14.555},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":17.222000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":8.3650000000000002},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":12.677}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":7.3090000000000002},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":15.252000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":18.597999999999999},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":11.462}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":20.242999999999999,"5min":19.175000000000001,"15min":18.997},"min":{"1min":12.217000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":44.826000000000001,"5min":61.445999999999998,"15min":61.445999999999998},"last":17.863},{"interface":"front","average":{"1min":22.859000000000002,"5min":20.625,"15min":20.253},"min":{"1min":13.35,"5min":11.833,"15min":11.833},"max":{"1min":38.460000000000001,"5min":63.518000000000001,"15min":63.518000000000001},"last":16.507000000000001}]}]},{"osd":7,"up_from":60,"seq":257698037796,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29000,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342840,"statfs":{"total":96636764160,"available":96607068160,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":7.0289999999999999},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":7.6459999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":5.7930000000000001},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":8.3300000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":9.7070000000000007},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":14.590999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":13.446},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":12.744}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":16.27},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":11.779999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":10.738},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":17.038}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":17.806999999999999},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":15.226000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542228,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":5.3689999999999998},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":7.7869999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.141,"5min":12.663,"15min":14.898999999999999},"min":{"1min":7.5880000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"max":{"1min":27.593,"5min":37.103999999999999,"15min":55.398000000000003},"last":11.717000000000001},{"interface":"front","average":{"1min":14.585000000000001,"5min":13.566000000000001,"15min":14.862},"min":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"max":{"1min":25.289000000000001,"5min":38.137,"15min":41.811999999999998},"last":14.000999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":16.024000000000001},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":8.4179999999999993}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":16.896999999999998},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":6.9710000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":19.532,"5min":15.497,"15min":22.34},"min":{"1min":10.414,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":31.209,"5min":31.209,"15min":117.10299999999999},"last":17.693000000000001},{"interface":"front","average":{"1min":20.265999999999998,"5min":17.077000000000002,"15min":24.907},"min":{"1min":6.9480000000000004,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":32.838999999999999,"5min":32.838999999999999,"15min":111.874},"last":13.02}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":12.294,"5min":11.125,"15min":15.733000000000001},"min":{"1min":7.5800000000000001,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":19.867999999999999,"5min":22.010000000000002,"15min":39.886000000000003},"last":11.343},{"interface":"front","average":{"1min":19.161999999999999,"5min":16.074000000000002,"15min":21.274000000000001},"min":{"1min":9.3499999999999996,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":28.93,"5min":30.199999999999999,"15min":60.530999999999999},"last":14.957000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":19.617000000000001,"5min":20.466000000000001,"15min":20.608000000000001},"min":{"1min":12.303000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":28.280999999999999,"5min":31.757999999999999,"15min":31.757999999999999},"last":18.463000000000001},{"interface":"front","average":{"1min":15.759,"5min":15.573,"15min":15.542},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":29.145,"5min":29.145,"15min":29.145},"last":10.760999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542228,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":13.714},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":5.4870000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":18.844999999999999},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":18.23}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":9.1959999999999997},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":10.952999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":15.298},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":10.256}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":14.382},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":20.396999999999998}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":17.597999999999999},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":16.234999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":14.670999999999999,"15min":14.456},"min":{"1min":7.3760000000000003,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":52.643999999999998,"5min":52.643999999999998,"15min":52.643999999999998},"last":13.132},{"interface":"front","average":{"1min":18.873999999999999,"5min":18.574999999999999,"15min":18.524999999999999},"min":{"1min":10.476000000000001,"5min":10.476000000000001,"15min":10.476000000000001},"max":{"1min":55.177999999999997,"5min":55.177999999999997,"15min":55.177999999999997},"last":16.806999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542228,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":9.2379999999999995},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":8.2040000000000006}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":10.613},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":19.117999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":15.256},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":7.3520000000000003}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":8.9559999999999995},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":17.335999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":25.759},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":11.387}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":21.225000000000001},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":14.881}]},{"osd":7,"last update":"Wed Dec 6 09:27:57 2023","interfaces":[{"interface":"back","average":{"1min":18.491,"5min":19.448,"15min":19.608000000000001},"min":{"1min":12.093,"5min":11.547000000000001,"15min":11.547000000000001},"max":{"1min":26.370999999999999,"5min":32.191000000000003,"15min":32.191000000000003},"last":16.556999999999999},{"interface":"front","average":{"1min":18.702999999999999,"5min":19.315000000000001,"15min":19.417000000000002},"min":{"1min":10.683,"5min":10.683,"15min":10.683},"max":{"1min":35.384999999999998,"5min":36.454000000000001,"15min":36.454000000000001},"last":20.344000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":5.3440000000000003},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":11.266999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":12.494},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":6.548}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":7.8410000000000002},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":13.173999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":17.021999999999998},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":8.4580000000000002}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":10.134},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":18.753}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":13.786},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":17.728999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:27:59 2023","interfaces":[{"interface":"back","average":{"1min":18.797000000000001,"5min":17.757000000000001,"15min":17.582999999999998},"min":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":32.920000000000002,"5min":32.920000000000002,"15min":32.920000000000002},"last":14.449999999999999},{"interface":"front","average":{"1min":20.239999999999998,"5min":18.984000000000002,"15min":18.774999999999999},"min":{"1min":11.968,"5min":11.663,"15min":11.663},"max":{"1min":41.348999999999997,"5min":41.348999999999997,"15min":41.348999999999997},"last":15.146000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:28:45.741 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.741 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.741 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.742 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.742 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.742 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.742 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-06T09:22:02.068230+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=22, tm_sec=2, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.742 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-06T09:21:55.261571+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=9, tm_min=28, tm_sec=37, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T09:28:45.742 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-06T09:28:48.418 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:26.396 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.420 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:26.399 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.438 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:26.416 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.460 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:20.774 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.464 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:20.779 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.466 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:06.813 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.472 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:06.817 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.478 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:20.793 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:28:48.487 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:06.833 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:05.743 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T09:29:06.321 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-12-06T09:29:06.322 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-12-06T09:29:06.335 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2023-12-06T09:29:05.993984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225292,"kb_used_data":8196,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754749428,"statfs":{"total":773094113280,"available":772863414272,"internally_reserved":0,"allocated":8392704,"data_stored":3782656,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":15557,"internal_metadata":222217019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002829"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:46.991540+0000","last_change":"2023-12-06T09:28:46.997787+0000","last_active":"2023-12-06T09:28:46.991540+0000","last_peered":"2023-12-06T09:28:46.991540+0000","last_clean":"2023-12-06T09:28:46.991540+0000","last_became_active":"2023-12-06T09:25:56.851821+0000","last_became_peered":"2023-12-06T09:25:56.851821+0000","last_unstale":"2023-12-06T09:28:46.991540+0000","last_undegraded":"2023-12-06T09:28:46.991540+0000","last_fullsized":"2023-12-06T09:28:46.991540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:46.987497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:46.987497+0000","last_clean_scrub_stamp":"2023-12-06T09:28:46.987497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:09:43.710548+0000","scrub_duration":0.33921192999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":36,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:49.193420+0000","last_change":"2023-12-06T09:28:49.198447+0000","last_active":"2023-12-06T09:28:49.193420+0000","last_peered":"2023-12-06T09:28:49.193420+0000","last_clean":"2023-12-06T09:28:49.193420+0000","last_became_active":"2023-12-06T09:28:08.034176+0000","last_became_peered":"2023-12-06T09:28:08.034176+0000","last_unstale":"2023-12-06T09:28:49.193420+0000","last_undegraded":"2023-12-06T09:28:49.193420+0000","last_fullsized":"2023-12-06T09:28:49.193420+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:49.187470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:49.187470+0000","last_clean_scrub_stamp":"2023-12-06T09:28:49.187470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:15:38.600988+0000","scrub_duration":0.19270624,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:45.883926+0000","last_change":"2023-12-06T09:28:45.884624+0000","last_active":"2023-12-06T09:28:45.883926+0000","last_peered":"2023-12-06T09:28:45.883926+0000","last_clean":"2023-12-06T09:28:45.883926+0000","last_became_active":"2023-12-06T09:28:07.987971+0000","last_became_peered":"2023-12-06T09:28:07.987971+0000","last_unstale":"2023-12-06T09:28:45.883926+0000","last_undegraded":"2023-12-06T09:28:45.883926+0000","last_fullsized":"2023-12-06T09:28:45.883926+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:45.882610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:45.882610+0000","last_clean_scrub_stamp":"2023-12-06T09:28:45.882610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:03:37.213532+0000","scrub_duration":0.14310082399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:41.882791+0000","last_change":"2023-12-06T09:28:41.889541+0000","last_active":"2023-12-06T09:28:41.882791+0000","last_peered":"2023-12-06T09:28:41.882791+0000","last_clean":"2023-12-06T09:28:41.882791+0000","last_became_active":"2023-12-06T09:28:06.895239+0000","last_became_peered":"2023-12-06T09:28:06.895239+0000","last_unstale":"2023-12-06T09:28:41.882791+0000","last_undegraded":"2023-12-06T09:28:41.882791+0000","last_fullsized":"2023-12-06T09:28:41.882791+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:41.878810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:41.878810+0000","last_clean_scrub_stamp":"2023-12-06T09:28:41.878810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:27:42.735018+0000","scrub_duration":0.30635094800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:48.833815+0000","last_change":"2023-12-06T09:28:48.834537+0000","last_active":"2023-12-06T09:28:48.833815+0000","last_peered":"2023-12-06T09:28:48.833815+0000","last_clean":"2023-12-06T09:28:48.833815+0000","last_became_active":"2023-12-06T09:28:18.778465+0000","last_became_peered":"2023-12-06T09:28:18.778465+0000","last_unstale":"2023-12-06T09:28:48.833815+0000","last_undegraded":"2023-12-06T09:28:48.833815+0000","last_fullsized":"2023-12-06T09:28:48.833815+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:48.832451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:48.832451+0000","last_clean_scrub_stamp":"2023-12-06T09:28:48.832451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:49:42.329664+0000","scrub_duration":0.12582162799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:48.627816+0000","last_change":"2023-12-06T09:28:48.628507+0000","last_active":"2023-12-06T09:28:48.627816+0000","last_peered":"2023-12-06T09:28:48.627816+0000","last_clean":"2023-12-06T09:28:48.627816+0000","last_became_active":"2023-12-06T09:28:00.963094+0000","last_became_peered":"2023-12-06T09:28:00.963094+0000","last_unstale":"2023-12-06T09:28:48.627816+0000","last_undegraded":"2023-12-06T09:28:48.627816+0000","last_fullsized":"2023-12-06T09:28:48.627816+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:48.626447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:48.626447+0000","last_clean_scrub_stamp":"2023-12-06T09:28:48.626447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:41:33.437070+0000","scrub_duration":0.040134086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:44.973035+0000","last_change":"2023-12-06T09:28:44.979489+0000","last_active":"2023-12-06T09:28:44.973035+0000","last_peered":"2023-12-06T09:28:44.973035+0000","last_clean":"2023-12-06T09:28:44.973035+0000","last_became_active":"2023-12-06T09:24:37.197397+0000","last_became_peered":"2023-12-06T09:24:37.197397+0000","last_unstale":"2023-12-06T09:28:44.973035+0000","last_undegraded":"2023-12-06T09:28:44.973035+0000","last_fullsized":"2023-12-06T09:28:44.973035+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:44.969231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:44.969231+0000","last_clean_scrub_stamp":"2023-12-06T09:28:44.969231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T21:19:57.302168+0000","scrub_duration":0.237155374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":44,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:50.782447+0000","last_change":"2023-12-06T09:28:50.783131+0000","last_active":"2023-12-06T09:28:50.782447+0000","last_peered":"2023-12-06T09:28:50.782447+0000","last_clean":"2023-12-06T09:28:50.782447+0000","last_became_active":"2023-12-06T09:28:18.634888+0000","last_became_peered":"2023-12-06T09:28:18.634888+0000","last_unstale":"2023-12-06T09:28:50.782447+0000","last_undegraded":"2023-12-06T09:28:50.782447+0000","last_fullsized":"2023-12-06T09:28:50.782447+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T09:28:50.781089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T09:28:50.781089+0000","last_clean_scrub_stamp":"2023-12-06T09:28:50.781089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:02:41.071206+0000","scrub_duration":0.088251687999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"127'78","reported_seq":255,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T09:28:44.846382+0000","last_change":"2023-12-06T09:28:44.853925+0000","last_active":"2023-12-06T09:28:44.846382+0000","last_peered":"2023-12-06T09:28:44.846382+0000","last_clean":"2023-12-06T09:28:44.846382+0000","last_became_active":"2023-12-06T09:24:37.254747+0000","last_became_peered":"2023-12-06T09:24:37.254747+0000","last_unstale":"2023-12-06T09:28:44.846382+0000","last_undegraded":"2023-12-06T09:28:44.846382+0000","last_fullsized":"2023-12-06T09:28:44.846382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'78","last_scrub_stamp":"2023-12-06T09:28:44.842556+0000","last_deep_scrub":"127'78","last_deep_scrub_stamp":"2023-12-06T09:28:44.842556+0000","last_clean_scrub_stamp":"2023-12-06T09:28:44.842556+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T21:03:24.962869+0000","scrub_duration":0.54188983800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509529,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":9.9629999999999992,"15min":14.664},"min":{"1min":5.2839999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"max":{"1min":47.256999999999998,"5min":47.256999999999998,"15min":49.691000000000003},"last":8.5009999999999994},{"interface":"front","average":{"1min":13.01,"5min":9.5990000000000002,"15min":14.693},"min":{"1min":7.2370000000000001,"5min":4.0709999999999997,"15min":4.0709999999999997},"max":{"1min":49.240000000000002,"5min":49.240000000000002,"15min":51.530000000000001},"last":7.6020000000000003}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.193,"5min":11.064,"15min":12.064},"min":{"1min":7.8979999999999997,"5min":5.0359999999999996,"15min":5.0359999999999996},"max":{"1min":32.238,"5min":40.991,"15min":40.991},"last":7.9269999999999996},{"interface":"front","average":{"1min":14.849,"5min":13.257999999999999,"15min":14.43},"min":{"1min":6.3220000000000001,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":34.798999999999999,"5min":35.850000000000001,"15min":35.850000000000001},"last":13.098000000000001}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.550000000000001,"5min":13.074,"15min":14.760999999999999},"min":{"1min":8.3520000000000003,"5min":6.9749999999999996,"15min":6.9749999999999996},"max":{"1min":52.393999999999998,"5min":52.393999999999998,"15min":52.393999999999998},"last":16.449000000000002},{"interface":"front","average":{"1min":17.884,"5min":15.178000000000001,"15min":17.257999999999999},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":55.865000000000002,"5min":55.865000000000002,"15min":55.865000000000002},"last":11.144}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":18.114000000000001,"5min":13.733000000000001,"15min":14.787000000000001},"min":{"1min":9.3699999999999992,"5min":5.4710000000000001,"15min":5.4710000000000001},"max":{"1min":38.542000000000002,"5min":38.542000000000002,"15min":38.542000000000002},"last":10.506},{"interface":"front","average":{"1min":17.975000000000001,"5min":15.977,"15min":15.728999999999999},"min":{"1min":9.7750000000000004,"5min":7.8550000000000004,"15min":7.7880000000000003},"max":{"1min":43.209000000000003,"5min":43.209000000000003,"15min":43.209000000000003},"last":15.048}]},{"osd":5,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":20.209,"5min":15.25,"15min":23.692},"min":{"1min":9.1579999999999995,"5min":5.9900000000000002,"15min":5.9900000000000002},"max":{"1min":39.524999999999999,"5min":39.524999999999999,"15min":190.821},"last":14.396000000000001},{"interface":"front","average":{"1min":18.021999999999998,"5min":16.422999999999998,"15min":23.901},"min":{"1min":9.734,"5min":6.1950000000000003,"15min":6.1950000000000003},"max":{"1min":36.898000000000003,"5min":44.625999999999998,"15min":191.86199999999999},"last":13.827}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":21.241,"5min":17.754000000000001,"15min":23.997},"min":{"1min":10.644,"5min":8.0210000000000008,"15min":8.0210000000000008},"max":{"1min":44.152000000000001,"5min":44.152000000000001,"15min":166.45400000000001},"last":10.199999999999999},{"interface":"front","average":{"1min":20.068000000000001,"5min":18.210000000000001,"15min":24.085999999999999},"min":{"1min":7.6950000000000003,"5min":7.6239999999999997,"15min":7.6239999999999997},"max":{"1min":37.597999999999999,"5min":48.956000000000003,"15min":117.58499999999999},"last":18.324000000000002}]},{"osd":7,"last update":"Wed Dec 6 09:29:00 2023","interfaces":[{"interface":"back","average":{"1min":17.992000000000001,"5min":22.501999999999999,"15min":23.288},"min":{"1min":10.516999999999999,"5min":10.516999999999999,"15min":10.516999999999999},"max":{"1min":27.751000000000001,"5min":52.109000000000002,"15min":52.109000000000002},"last":11.816000000000001},{"interface":"front","average":{"1min":18.434999999999999,"5min":21.940999999999999,"15min":22.657},"min":{"1min":11.339,"5min":11.01,"15min":11.01},"max":{"1min":27.469999999999999,"5min":56.914000000000001,"15min":56.914000000000001},"last":17.300999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":8.7189999999999994,"5min":9.2929999999999993,"15min":8.2550000000000008},"min":{"1min":4.1699999999999999,"5min":3.6549999999999998,"15min":3.6549999999999998},"max":{"1min":17.614999999999998,"5min":38.481000000000002,"15min":38.481000000000002},"last":11.079000000000001},{"interface":"front","average":{"1min":10.061999999999999,"5min":11.266,"15min":11.993},"min":{"1min":5.4409999999999998,"5min":5.4409999999999998,"15min":5.4409999999999998},"max":{"1min":20.661000000000001,"5min":37.134999999999998,"15min":37.134999999999998},"last":7.9630000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":13.208,"5min":12.949999999999999,"15min":14.568},"min":{"1min":6.327,"5min":5.7880000000000003,"15min":5.7880000000000003},"max":{"1min":32.789000000000001,"5min":32.789000000000001,"15min":52.530000000000001},"last":10.460000000000001},{"interface":"front","average":{"1min":18.43,"5min":16.625,"15min":18.187000000000001},"min":{"1min":10.759,"5min":9.6699999999999999,"15min":9.2240000000000002},"max":{"1min":32.176000000000002,"5min":32.176000000000002,"15min":54.338000000000001},"last":11.738}]},{"osd":3,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":12.393000000000001,"15min":11.430999999999999},"min":{"1min":6.5640000000000001,"5min":5.5899999999999999,"15min":5.5899999999999999},"max":{"1min":32.860999999999997,"5min":32.860999999999997,"15min":32.860999999999997},"last":16.245999999999999},{"interface":"front","average":{"1min":10.214,"5min":10.563000000000001,"15min":12.558},"min":{"1min":5.6740000000000004,"5min":5.4269999999999996,"15min":5.4269999999999996},"max":{"1min":26.428000000000001,"5min":28.669,"15min":28.669},"last":15.326000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":15.371,"5min":14.039,"15min":14.821},"min":{"1min":8.1170000000000009,"5min":7.2130000000000001,"15min":7.2130000000000001},"max":{"1min":34.479999999999997,"5min":54.32,"15min":54.32},"last":17.260000000000002},{"interface":"front","average":{"1min":19.466000000000001,"5min":17.052,"15min":17.603000000000002},"min":{"1min":9.2910000000000004,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":42.258000000000003,"5min":51.950000000000003,"15min":51.950000000000003},"last":12.359999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":12.628,"5min":12.32,"15min":14.917999999999999},"min":{"1min":6.9210000000000003,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":25.553999999999998,"5min":48.563000000000002,"15min":56.491},"last":9.75},{"interface":"front","average":{"1min":16.669,"5min":15.77,"15min":18.765000000000001},"min":{"1min":10.634,"5min":7.1920000000000002,"15min":7.1920000000000002},"max":{"1min":42.469000000000001,"5min":47.328000000000003,"15min":75.540999999999997},"last":20.797000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:07 2023","interfaces":[{"interface":"back","average":{"1min":19.210000000000001,"5min":17.431000000000001,"15min":18.103999999999999},"min":{"1min":12.612,"5min":10.010999999999999,"15min":10.010999999999999},"max":{"1min":41.619999999999997,"5min":41.619999999999997,"15min":41.619999999999997},"last":13.071999999999999},{"interface":"front","average":{"1min":14.885,"5min":13.641999999999999,"15min":14.332000000000001},"min":{"1min":7.5209999999999999,"5min":7.0190000000000001,"15min":7.0190000000000001},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":14.686}]},{"osd":7,"last update":"Wed Dec 6 09:28:01 2023","interfaces":[{"interface":"back","average":{"1min":20.898,"5min":18.292999999999999,"15min":17.859000000000002},"min":{"1min":12.004,"5min":12.004,"15min":12.004},"max":{"1min":45.728000000000002,"5min":45.728000000000002,"15min":45.728000000000002},"last":18.324999999999999},{"interface":"front","average":{"1min":23.457999999999998,"5min":20.518000000000001,"15min":20.027999999999999},"min":{"1min":13.983000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"max":{"1min":55.488999999999997,"5min":55.488999999999997,"15min":55.488999999999997},"last":18.946000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":11.211,"15min":11.220000000000001},"min":{"1min":4.7409999999999997,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":38.741,"5min":65.808000000000007,"15min":65.808000000000007},"last":13.284000000000001},{"interface":"front","average":{"1min":10.846,"5min":12.067,"15min":11.574999999999999},"min":{"1min":5.9530000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"max":{"1min":17.824000000000002,"5min":53.106999999999999,"15min":53.106999999999999},"last":7.8490000000000002}]},{"osd":2,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":15.000999999999999,"5min":13.448,"15min":13.206},"min":{"1min":6.9589999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":43.972999999999999,"5min":49.652999999999999,"15min":49.652999999999999},"last":16.143999999999998},{"interface":"front","average":{"1min":16.279,"5min":13.632999999999999,"15min":14.170999999999999},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":46.619,"5min":47.484000000000002,"15min":47.484000000000002},"last":9.2140000000000004}]},{"osd":3,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":11.282,"5min":11.311999999999999,"15min":19.274000000000001},"min":{"1min":6.8659999999999997,"5min":6.0060000000000002,"15min":6.0060000000000002},"max":{"1min":25.553000000000001,"5min":41.765999999999998,"15min":205.096},"last":5.9269999999999996},{"interface":"front","average":{"1min":15.375999999999999,"5min":15.662000000000001,"15min":21.562999999999999},"min":{"1min":9.2750000000000004,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":25.690999999999999,"5min":52.024999999999999,"15min":182.74000000000001},"last":14.692}]},{"osd":4,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":17.710000000000001,"5min":13.898999999999999,"15min":14.202},"min":{"1min":7.6559999999999997,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":36.786000000000001,"5min":45.112000000000002,"15min":45.112000000000002},"last":6.7789999999999999},{"interface":"front","average":{"1min":18.172999999999998,"5min":17.152000000000001,"15min":16.896000000000001},"min":{"1min":11.359999999999999,"5min":7.1470000000000002,"15min":7.1470000000000002},"max":{"1min":35.604999999999997,"5min":52.655999999999999,"15min":52.655999999999999},"last":17.015000000000001}]},{"osd":5,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.451000000000001,"5min":16.219000000000001,"15min":15.451000000000001},"min":{"1min":7.3570000000000002,"5min":6.6600000000000001,"15min":6.6600000000000001},"max":{"1min":39.840000000000003,"5min":43.088000000000001,"15min":43.088000000000001},"last":11.973000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.719000000000001,"15min":16.920999999999999},"min":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"max":{"1min":39.177,"5min":45.722000000000001,"15min":45.722000000000001},"last":10.872}]},{"osd":6,"last update":"Wed Dec 6 09:28:05 2023","interfaces":[{"interface":"back","average":{"1min":18.788,"5min":18.852,"15min":20.341000000000001},"min":{"1min":10.009,"5min":9.2309999999999999,"15min":7.5640000000000001},"max":{"1min":32.424999999999997,"5min":53.921999999999997,"15min":78.338999999999999},"last":17.800000000000001},{"interface":"front","average":{"1min":20.670999999999999,"5min":17.702999999999999,"15min":20.219999999999999},"min":{"1min":9.7010000000000005,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":34.250999999999998,"5min":50.877000000000002,"15min":80.850999999999999},"last":12.676}]},{"osd":7,"last update":"Wed Dec 6 09:29:01 2023","interfaces":[{"interface":"back","average":{"1min":15.382,"5min":18.469999999999999,"15min":18.762},"min":{"1min":10.146000000000001,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":27.542000000000002,"5min":61.445999999999998,"15min":61.445999999999998},"last":10.146000000000001},{"interface":"front","average":{"1min":19.134,"5min":20.437999999999999,"15min":20.190000000000001},"min":{"1min":13.808,"5min":11.833,"15min":11.833},"max":{"1min":35.259999999999998,"5min":63.518000000000001,"15min":63.518000000000001},"last":13.92}]}]},{"osd":7,"up_from":60,"seq":257698037801,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28992,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94342848,"statfs":{"total":96636764160,"available":96607076352,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4395,"internal_metadata":28634837},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":10.144,"5min":11.616,"15min":11.862},"min":{"1min":4.585,"5min":4.452,"15min":4.452},"max":{"1min":27.565000000000001,"5min":55.622,"15min":55.622},"last":8.6430000000000007},{"interface":"front","average":{"1min":15.914999999999999,"5min":12.853999999999999,"15min":12.343999999999999},"min":{"1min":6.7329999999999997,"5min":6.5650000000000004,"15min":6.5650000000000004},"max":{"1min":60.448999999999998,"5min":60.448999999999998,"15min":60.448999999999998},"last":13.050000000000001}]},{"osd":1,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":14.069000000000001,"5min":14.94,"15min":15.085000000000001},"min":{"1min":7.476,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":28.062999999999999,"5min":48.959000000000003,"15min":48.959000000000003},"last":15.843},{"interface":"front","average":{"1min":12.721,"5min":13.74,"15min":13.91},"min":{"1min":6.2009999999999996,"5min":6.2009999999999996,"15min":6.2009999999999996},"max":{"1min":19.106999999999999,"5min":38.232999999999997,"15min":38.232999999999997},"last":11.292999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":16.710000000000001,"5min":13.098000000000001,"15min":12.496},"min":{"1min":9.6470000000000002,"5min":5.952,"15min":5.952},"max":{"1min":23.989999999999998,"5min":23.989999999999998,"15min":23.989999999999998},"last":12.134},{"interface":"front","average":{"1min":17.739000000000001,"5min":15.82,"15min":15.5},"min":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"max":{"1min":29.916,"5min":48.240000000000002,"15min":48.240000000000002},"last":19.498999999999999}]},{"osd":3,"last update":"Wed Dec 6 09:28:04 2023","interfaces":[{"interface":"back","average":{"1min":12.288,"5min":13.311999999999999,"15min":13.481999999999999},"min":{"1min":6.181,"5min":6.181,"15min":6.181},"max":{"1min":25.428999999999998,"5min":54.603999999999999,"15min":54.603999999999999},"last":15.536},{"interface":"front","average":{"1min":16.542000000000002,"5min":20.274999999999999,"15min":20.896999999999998},"min":{"1min":7.6550000000000002,"5min":7.6550000000000002,"15min":7.6550000000000002},"max":{"1min":26.966000000000001,"5min":52.719000000000001,"15min":52.719000000000001},"last":12.614000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":18.364999999999998,"5min":15.708,"15min":15.265000000000001},"min":{"1min":10.755000000000001,"5min":7.2329999999999997,"15min":7.2329999999999997},"max":{"1min":30.542000000000002,"5min":32.390999999999998,"15min":32.390999999999998},"last":17.344999999999999},{"interface":"front","average":{"1min":15.619999999999999,"5min":14.058,"15min":13.798},"min":{"1min":8.1790000000000003,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":28.826000000000001,"5min":30.219999999999999,"15min":30.219999999999999},"last":10.651}]},{"osd":5,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":19.294,"15min":19.52},"min":{"1min":10.944000000000001,"5min":10.944000000000001,"15min":10.944000000000001},"max":{"1min":26.657,"5min":52.197000000000003,"15min":52.197000000000003},"last":22.181000000000001},{"interface":"front","average":{"1min":19.745999999999999,"5min":17.837,"15min":17.518999999999998},"min":{"1min":13.196999999999999,"5min":7.3419999999999996,"15min":7.3419999999999996},"max":{"1min":31.157,"5min":38.877000000000002,"15min":38.877000000000002},"last":24.451000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:00 2023","interfaces":[{"interface":"back","average":{"1min":20.515999999999998,"5min":19.488,"15min":19.315999999999999},"min":{"1min":15.340999999999999,"5min":12.494999999999999,"15min":12.494999999999999},"max":{"1min":32.341000000000001,"5min":39.508000000000003,"15min":39.508000000000003},"last":21.187999999999999},{"interface":"front","average":{"1min":14.858000000000001,"5min":14.901,"15min":14.907999999999999},"min":{"1min":7.8979999999999997,"5min":7.8979999999999997,"15min":7.8979999999999997},"max":{"1min":22.664000000000001,"5min":24.553999999999998,"15min":24.553999999999998},"last":14.958}]}]},{"osd":6,"up_from":14,"seq":60129542233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":11.343,"5min":10.061,"15min":11.465999999999999},"min":{"1min":5.7050000000000001,"5min":3.484,"15min":3.484},"max":{"1min":27.433,"5min":39.012999999999998,"15min":45.506},"last":14.550000000000001},{"interface":"front","average":{"1min":13.407999999999999,"5min":10.581,"15min":12.597},"min":{"1min":6.6779999999999999,"5min":5.1120000000000001,"15min":5.1120000000000001},"max":{"1min":26.370000000000001,"5min":40.880000000000003,"15min":46.450000000000003},"last":6.9089999999999998}]},{"osd":1,"last update":"Wed Dec 6 09:29:02 2023","interfaces":[{"interface":"back","average":{"1min":12.853999999999999,"5min":12.742000000000001,"15min":14.688000000000001},"min":{"1min":7.3570000000000002,"5min":6.391,"15min":5.0380000000000003},"max":{"1min":21.864999999999998,"5min":30.434999999999999,"15min":55.398000000000003},"last":7.6360000000000001},{"interface":"front","average":{"1min":12.117000000000001,"5min":13.211,"15min":14.635999999999999},"min":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"max":{"1min":19.588999999999999,"5min":31.419,"15min":41.811999999999998},"last":16.099}]},{"osd":2,"last update":"Wed Dec 6 09:28:15 2023","interfaces":[{"interface":"back","average":{"1min":14.363,"5min":13.782999999999999,"15min":19.853999999999999},"min":{"1min":8.8059999999999992,"5min":5.8920000000000003,"15min":5.1420000000000003},"max":{"1min":27.669,"5min":27.669,"15min":208.93799999999999},"last":17.939},{"interface":"front","average":{"1min":15.603,"5min":13.779,"15min":23.222999999999999},"min":{"1min":8.3369999999999997,"5min":6.0119999999999996,"15min":6.0119999999999996},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":208.57400000000001},"last":9.3629999999999995}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":18.257999999999999,"5min":15.590999999999999,"15min":17.213000000000001},"min":{"1min":8.7119999999999997,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":33.328000000000003,"5min":33.328000000000003,"15min":43.276000000000003},"last":17.027000000000001},{"interface":"front","average":{"1min":12.220000000000001,"5min":10.215,"15min":13.891},"min":{"1min":6.7290000000000001,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":23.039999999999999,"5min":23.039999999999999,"15min":41.493000000000002},"last":11.443}]},{"osd":4,"last update":"Wed Dec 6 09:29:02 2023","interfaces":[{"interface":"back","average":{"1min":16.805,"5min":15.896000000000001,"15min":21.742999999999999},"min":{"1min":11.166,"5min":7.4139999999999997,"15min":7.4139999999999997},"max":{"1min":22.635000000000002,"5min":31.209,"15min":117.10299999999999},"last":18.812000000000001},{"interface":"front","average":{"1min":15.516,"5min":17.16,"15min":24.02},"min":{"1min":9.8439999999999994,"5min":5.5990000000000002,"15min":5.5990000000000002},"max":{"1min":21.677,"5min":32.838999999999999,"15min":111.874},"last":10.484999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:29:02 2023","interfaces":[{"interface":"back","average":{"1min":16.079999999999998,"5min":12.073,"15min":15.603},"min":{"1min":6.8739999999999997,"5min":5.7359999999999998,"15min":5.7359999999999998},"max":{"1min":66.366,"5min":66.366,"15min":66.366},"last":13.076000000000001},{"interface":"front","average":{"1min":19.762,"5min":16.879000000000001,"15min":21},"min":{"1min":9.9199999999999999,"5min":9.0009999999999994,"15min":9.0009999999999994},"max":{"1min":68.090999999999994,"5min":68.090999999999994,"15min":68.090999999999994},"last":12.167999999999999}]},{"osd":7,"last update":"Wed Dec 6 09:29:02 2023","interfaces":[{"interface":"back","average":{"1min":17.335000000000001,"5min":19.797000000000001,"15min":20.385000000000002},"min":{"1min":12.675000000000001,"5min":12.303000000000001,"15min":12.303000000000001},"max":{"1min":25.791,"5min":31.757999999999999,"15min":31.757999999999999},"last":19.794},{"interface":"front","average":{"1min":13.891999999999999,"5min":15.246,"15min":15.433},"min":{"1min":8.0419999999999998,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":27.050999999999998,"5min":29.145,"15min":29.145},"last":13.785}]}]},{"osd":5,"up_from":14,"seq":60129542233,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1073152,"data_stored":771212,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":11.045999999999999,"5min":9.7959999999999994,"15min":12.846},"min":{"1min":4.7530000000000001,"5min":4.0659999999999998,"15min":4.0659999999999998},"max":{"1min":38.012,"5min":40.682000000000002,"15min":45.491},"last":4.9029999999999996},{"interface":"front","average":{"1min":9.9480000000000004,"5min":9.9619999999999997,"15min":12.630000000000001},"min":{"1min":4.8150000000000004,"5min":4.0129999999999999,"15min":4.0129999999999999},"max":{"1min":36.920999999999999,"5min":39.033999999999999,"15min":49.991999999999997},"last":11.667999999999999}]},{"osd":1,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.699999999999999,"5min":11.345000000000001,"15min":20.478999999999999},"min":{"1min":5.8579999999999997,"5min":5.4619999999999997,"15min":4.6710000000000003},"max":{"1min":21.309000000000001,"5min":39.326999999999998,"15min":220.13900000000001},"last":19.486999999999998},{"interface":"front","average":{"1min":10.308,"5min":12.276,"15min":27.920000000000002},"min":{"1min":6.6589999999999998,"5min":5.9059999999999997,"15min":5.9059999999999997},"max":{"1min":16.366,"5min":74.992999999999995,"15min":344.79599999999999},"last":10.882999999999999}]},{"osd":2,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":13.773,"5min":12.913,"15min":13.388999999999999},"min":{"1min":6.391,"5min":6.1710000000000003,"15min":6.1710000000000003},"max":{"1min":56.463000000000001,"5min":56.463000000000001,"15min":56.463000000000001},"last":13.593999999999999},{"interface":"front","average":{"1min":16.224,"5min":13.613,"15min":15.130000000000001},"min":{"1min":8.4130000000000003,"5min":7.0599999999999996,"15min":7.0599999999999996},"max":{"1min":55.203000000000003,"5min":55.203000000000003,"15min":55.203000000000003},"last":7.3099999999999996}]},{"osd":3,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.624000000000001,"15min":14.210000000000001},"min":{"1min":6.1020000000000003,"5min":6.1020000000000003,"15min":6.1020000000000003},"max":{"1min":56.594000000000001,"5min":56.594000000000001,"15min":56.594000000000001},"last":14.259},{"interface":"front","average":{"1min":13.260999999999999,"5min":11.904,"15min":15.800000000000001},"min":{"1min":6.3630000000000004,"5min":5.3760000000000003,"15min":5.3760000000000003},"max":{"1min":55.229999999999997,"5min":55.229999999999997,"15min":55.229999999999997},"last":12.380000000000001}]},{"osd":4,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":12.289999999999999,"5min":12.975,"15min":19.132000000000001},"min":{"1min":6.8529999999999998,"5min":6.085,"15min":6.085},"max":{"1min":18.381,"5min":44.585000000000001,"15min":281.608},"last":15.032999999999999},{"interface":"front","average":{"1min":15.355,"5min":15.079000000000001,"15min":21.170000000000002},"min":{"1min":10.571,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":22.204000000000001,"5min":45.780999999999999,"15min":225.83600000000001},"last":18.059000000000001}]},{"osd":6,"last update":"Wed Dec 6 09:28:10 2023","interfaces":[{"interface":"back","average":{"1min":16.143999999999998,"5min":15.525,"15min":20.216999999999999},"min":{"1min":9.2219999999999995,"5min":8.1780000000000008,"15min":8.1780000000000008},"max":{"1min":35.011000000000003,"5min":35.011000000000003,"15min":198.667},"last":18.736999999999998},{"interface":"front","average":{"1min":17.225999999999999,"5min":16.042999999999999,"15min":21.591000000000001},"min":{"1min":8.5790000000000006,"5min":7.9429999999999996,"15min":7.9429999999999996},"max":{"1min":42.646999999999998,"5min":42.646999999999998,"15min":208.446},"last":15.789}]},{"osd":7,"last update":"Wed Dec 6 09:29:00 2023","interfaces":[{"interface":"back","average":{"1min":14.747,"5min":14.750999999999999,"15min":14.483000000000001},"min":{"1min":7.3140000000000001,"5min":6.5049999999999999,"15min":6.5049999999999999},"max":{"1min":22.457000000000001,"5min":52.643999999999998,"15min":52.643999999999998},"last":8.0820000000000007},{"interface":"front","average":{"1min":17.199000000000002,"5min":18.315000000000001,"15min":18.439},"min":{"1min":9.9830000000000005,"5min":9.9830000000000005,"15min":9.9830000000000005},"max":{"1min":23.584,"5min":55.177999999999997,"15min":55.177999999999997},"last":16.423999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1077248,"data_stored":771212,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.609999999999999,"5min":9.2129999999999992,"15min":10.653},"min":{"1min":5.4710000000000001,"5min":4.0010000000000003,"15min":4.0010000000000003},"max":{"1min":31.459,"5min":31.459,"15min":40.225999999999999},"last":8.5839999999999996},{"interface":"front","average":{"1min":12.404999999999999,"5min":10.359999999999999,"15min":11.509},"min":{"1min":5.3319999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"max":{"1min":31.234999999999999,"5min":31.234999999999999,"15min":31.234999999999999},"last":5.7009999999999996}]},{"osd":2,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":12.179,"5min":13.977,"15min":11.785},"min":{"1min":6.6509999999999998,"5min":5.6799999999999997,"15min":4.6509999999999998},"max":{"1min":30.559000000000001,"5min":162.74299999999999,"15min":162.74299999999999},"last":21.806000000000001},{"interface":"front","average":{"1min":17.271999999999998,"5min":17.59,"15min":16.167000000000002},"min":{"1min":10.021000000000001,"5min":6.798,"15min":6.415},"max":{"1min":31.259,"5min":164.911,"15min":164.911},"last":19.131}]},{"osd":3,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":15.015000000000001,"5min":12.984,"15min":19.170000000000002},"min":{"1min":6.7590000000000003,"5min":5.359,"15min":5.359},"max":{"1min":44.472000000000001,"5min":44.472000000000001,"15min":122.762},"last":7.9950000000000001},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.004,"15min":18.472000000000001},"min":{"1min":6.2919999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"max":{"1min":48.767000000000003,"5min":48.767000000000003,"15min":129.99100000000001},"last":10.683999999999999}]},{"osd":4,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":14.67,"5min":14.695,"15min":26.744},"min":{"1min":7.0739999999999998,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":55.831000000000003,"5min":55.831000000000003,"15min":359.46300000000002},"last":11.471},{"interface":"front","average":{"1min":15.205,"5min":14.449,"15min":25.652999999999999},"min":{"1min":6.2140000000000004,"5min":5.6139999999999999,"15min":5.6139999999999999},"max":{"1min":32.069000000000003,"5min":47.192,"15min":314.47699999999998},"last":12.263999999999999}]},{"osd":5,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.242000000000001,"15min":25.239999999999998},"min":{"1min":9.8900000000000006,"5min":7.8109999999999999,"15min":7.8109999999999999},"max":{"1min":46.366999999999997,"5min":94.649000000000001,"15min":152.97399999999999},"last":13.239000000000001},{"interface":"front","average":{"1min":17.795000000000002,"5min":16.902000000000001,"15min":23.486000000000001},"min":{"1min":9.7210000000000001,"5min":8.375,"15min":8.375},"max":{"1min":32.960000000000001,"5min":50.206000000000003,"15min":154.53100000000001},"last":16.526}]},{"osd":6,"last update":"Wed Dec 6 09:28:08 2023","interfaces":[{"interface":"back","average":{"1min":20.094999999999999,"5min":15.651,"15min":17.175999999999998},"min":{"1min":7.7380000000000004,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":49.881999999999998,"5min":49.881999999999998,"15min":49.881999999999998},"last":9.9730000000000008},{"interface":"front","average":{"1min":19.696000000000002,"5min":17.628,"15min":20.170999999999999},"min":{"1min":8.9879999999999995,"5min":8.7919999999999998,"15min":8.7919999999999998},"max":{"1min":52.569000000000003,"5min":52.569000000000003,"15min":52.569000000000003},"last":18.042000000000002}]},{"osd":7,"last update":"Wed Dec 6 09:28:58 2023","interfaces":[{"interface":"back","average":{"1min":23.132999999999999,"5min":20.137,"15min":19.837},"min":{"1min":9.5739999999999998,"5min":9.5739999999999998,"15min":9.5739999999999998},"max":{"1min":89.760999999999996,"5min":89.760999999999996,"15min":89.760999999999996},"last":19.946999999999999},{"interface":"front","average":{"1min":21.925000000000001,"5min":19.806000000000001,"15min":19.579999999999998},"min":{"1min":10.481999999999999,"5min":10.481999999999999,"15min":10.481999999999999},"max":{"1min":83.018000000000001,"5min":83.018000000000001,"15min":83.018000000000001},"last":17.384}]}]},{"osd":4,"up_from":13,"seq":55834574939,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":373372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":9.4909999999999997,"5min":10.586,"15min":13.255000000000001},"min":{"1min":4.4240000000000004,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":29.448,"5min":137.13999999999999,"15min":137.13999999999999},"last":4.899},{"interface":"front","average":{"1min":12.074999999999999,"5min":11.113,"15min":17.125},"min":{"1min":5.0039999999999996,"5min":4.0990000000000002,"15min":4.0990000000000002},"max":{"1min":30.018999999999998,"5min":151.34999999999999,"15min":199.268},"last":11.036}]},{"osd":1,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":13.977,"5min":12.266999999999999,"15min":17.876999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":28.957000000000001,"5min":45.438000000000002,"15min":92.817999999999998},"last":14.186},{"interface":"front","average":{"1min":10.302,"5min":12.233000000000001,"15min":18.699999999999999},"min":{"1min":5.6580000000000004,"5min":5.556,"15min":5.556},"max":{"1min":26.401,"5min":31.067,"15min":95.665999999999997},"last":9.9610000000000003}]},{"osd":2,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":15.417999999999999,"15min":22.356000000000002},"min":{"1min":7.0220000000000002,"5min":7.0220000000000002,"15min":6.7160000000000002},"max":{"1min":43.426000000000002,"5min":156.92500000000001,"15min":208.06},"last":11.704000000000001},{"interface":"front","average":{"1min":19.507999999999999,"5min":16.314,"15min":18.619},"min":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":6.915},"max":{"1min":48.792999999999999,"5min":147.18199999999999,"15min":147.18199999999999},"last":16.968}]},{"osd":3,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":15.266,"5min":13.006,"15min":17.824000000000002},"min":{"1min":6.6619999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":30.384,"5min":37.485999999999997,"15min":121.627},"last":7.6180000000000003},{"interface":"front","average":{"1min":18.013999999999999,"5min":16.34,"15min":18.170000000000002},"min":{"1min":10.977,"5min":6.5179999999999998,"15min":6.5179999999999998},"max":{"1min":32.774000000000001,"5min":43.918999999999997,"15min":74.602000000000004},"last":12.747}]},{"osd":5,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":16.545000000000002,"15min":18.739000000000001},"min":{"1min":9.4930000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"max":{"1min":35.880000000000003,"5min":42.881999999999998,"15min":71.759},"last":13.566000000000001},{"interface":"front","average":{"1min":16.228000000000002,"5min":13.933999999999999,"15min":16.631},"min":{"1min":6.8040000000000003,"5min":6.6399999999999997,"15min":6.6399999999999997},"max":{"1min":34.609999999999999,"5min":41.155999999999999,"15min":70.882999999999996},"last":8.8480000000000008}]},{"osd":6,"last update":"Wed Dec 6 09:28:11 2023","interfaces":[{"interface":"back","average":{"1min":17.905000000000001,"5min":15.68,"15min":19.623999999999999},"min":{"1min":9.1620000000000008,"5min":6.6639999999999997,"15min":6.6639999999999997},"max":{"1min":30.888999999999999,"5min":48.136000000000003,"15min":80.668999999999997},"last":18.802},{"interface":"front","average":{"1min":17.821999999999999,"5min":18.012,"15min":21.388000000000002},"min":{"1min":8.4700000000000006,"5min":7.0919999999999996,"15min":7.0919999999999996},"max":{"1min":30.475000000000001,"5min":46.707000000000001,"15min":72.796999999999997},"last":15.194000000000001}]},{"osd":7,"last update":"Wed Dec 6 09:28:59 2023","interfaces":[{"interface":"back","average":{"1min":16.239000000000001,"5min":17.504999999999999,"15min":17.498999999999999},"min":{"1min":10.731,"5min":8.4819999999999993,"15min":8.4819999999999993},"max":{"1min":24.001000000000001,"5min":32.920000000000002,"15min":32.920000000000002},"last":17.928000000000001},{"interface":"front","average":{"1min":18.960000000000001,"5min":19.042000000000002,"15min":18.794},"min":{"1min":12.073,"5min":11.663,"15min":11.663},"max":{"1min":29.190000000000001,"5min":41.348999999999997,"15min":41.348999999999997},"last":15.816000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T09:29:06.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-06T09:29:07.091 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-06T09:29:07.091 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-06T09:29:07.091 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-06T09:29:07.091 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:29:07.099 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:29:07.106+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86940) UID: 0 2023-12-06T09:29:07.100 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-12-06T09:29:07.106+0000 ac7d640 -1 osd.0 127 *** Got signal Terminated *** 2023-12-06T09:29:08.537 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:46.516 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.587 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:26.933 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.591 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:40.905 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.591 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:26.937 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.594 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:40.908 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.594 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:46.573 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.606 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:26.952 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.608 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:02:40.923 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:08.612 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:46.590 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.524 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:06.502 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.525 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:06.504 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.538 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:06.516 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.568 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:46.914 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.573 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:00.888 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.574 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:46.920 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.588 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:46.934 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.589 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:00.904 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:28.605 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:00.919 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.563 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:26.541 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.567 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:20.881 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.568 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:26.546 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.570 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:20.885 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.586 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:26.564 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.587 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:20.902 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.611 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:06.957 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.621 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:06.967 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:29:48.635 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:06.981 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.607 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:46.586 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.609 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:46.588 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.622 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:46.601 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.658 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:40.973 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.659 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:40.974 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.660 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:27.006 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.662 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:27.008 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.673 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:03:40.988 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:08.675 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:27.021 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.653 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:06.632 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.655 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:06.634 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.668 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:06.647 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.701 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:01.016 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.706 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:01.020 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.716 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:47.062 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.718 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:47.064 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.720 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:01.035 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:28.733 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:47.079 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:38.618 INFO:tasks.ceph.osd.0:Stopped 2023-12-06T09:30:38.619 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-06T09:30:38.619 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:30:38.629 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:30:38.634+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86943) UID: 0 2023-12-06T09:30:38.631 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-12-06T09:30:38.634+0000 ac7d640 -1 osd.1 129 *** Got signal Terminated *** 2023-12-06T09:30:39.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-06T09:30:45.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-06T09:30:48.696 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:26.674 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.698 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:26.676 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.719 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:26.698 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.751 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:21.066 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.753 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:21.068 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.757 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:07.103 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.760 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:07.106 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.769 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:21.084 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:48.781 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:07.127 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.726 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:36.704 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.730 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:36.707 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.742 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:36.720 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.777 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:31.091 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.777 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:17.123 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.779 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:31.094 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.782 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:17.128 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.793 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:31.107 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:30:58.798 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:17.143 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.753 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:46.731 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.757 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:46.736 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.770 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:46.749 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.805 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:41.120 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.806 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:27.151 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.807 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:41.122 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.812 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:27.158 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.818 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:41.132 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:08.822 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:27.168 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:09.126 INFO:tasks.ceph.osd.1:Stopped 2023-12-06T09:31:09.126 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-06T09:31:09.126 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:31:09.134 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:31:09.141+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86945) UID: 0 2023-12-06T09:31:09.135 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-12-06T09:31:09.142+0000 ac7d640 -1 osd.2 131 *** Got signal Terminated *** 2023-12-06T09:31:13.892 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:51.870 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.898 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:51.877 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.914 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:51.892 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.946 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:46.261 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.947 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:32.293 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.948 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:46.263 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.949 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:32.294 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.962 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:04:46.277 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:13.963 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:32.309 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.805 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:06.783 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.810 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:06.789 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.828 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:06.807 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.859 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:01.174 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.860 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:01.175 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.861 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:47.207 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.863 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:47.209 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.873 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:01.188 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:28.876 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:47.222 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:39.633 INFO:tasks.ceph.osd.2:Stopped 2023-12-06T09:31:39.634 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-06T09:31:39.634 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:31:39.642 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:31:39.649+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86950) UID: 0 2023-12-06T09:31:39.642 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-12-06T09:31:39.649+0000 ac7d640 -1 osd.3 133 *** Got signal Terminated *** 2023-12-06T09:31:45.736 INFO:tasks.ceph.osd.3:Stopped 2023-12-06T09:31:45.736 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-06T09:31:45.736 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:31:45.745 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:31:45.751+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86667) UID: 0 2023-12-06T09:31:45.748 INFO:tasks.ceph.osd.4.smithi132.stderr:2023-12-06T09:31:45.752+0000 ac7d640 -1 osd.4 135 *** Got signal Terminated *** 2023-12-06T09:31:48.969 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:26.948 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:48.973 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:26.951 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:48.987 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:26.966 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:49.018 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:21.332 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:49.019 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:21.333 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:49.020 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:07.366 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:49.022 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:07.368 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:49.034 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:21.349 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:31:49.045 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:07.391 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.910 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:46.888 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.913 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:46.892 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.930 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:46.908 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.962 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:41.277 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.963 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:41.278 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.964 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:27.311 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.969 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:27.315 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.978 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:05:41.292 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:08.987 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:27.333 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:28.964 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:06.942 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:28.968 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:06.946 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:28.983 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:06.962 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:29.012 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:47.358 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:29.013 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:01.328 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:29.017 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:01.331 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:29.018 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:47.364 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:29.031 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:01.346 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:29.032 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:47.378 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.013 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:26.991 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.015 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:26.994 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.033 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:27.011 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.060 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:21.374 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.066 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:21.381 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.069 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:07.415 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.070 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:07.416 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.081 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:21.395 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:32:49.083 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:07.429 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.185 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:47.163 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.189 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:47.168 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.203 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:47.182 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.233 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:27.579 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.234 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:41.548 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.236 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:41.551 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.236 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:27.582 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.250 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:27.596 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:09.256 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:06:41.571 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.136 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:07.115 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.138 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:07.117 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.152 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:07.131 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.181 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:01.495 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.183 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:01.498 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.190 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:47.536 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.192 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:47.538 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.197 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:01.512 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:29.206 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:47.552 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:47.766 INFO:tasks.ceph.osd.4:Stopped 2023-12-06T09:33:47.766 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-06T09:33:47.766 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:33:47.774 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:33:47.781+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86670) UID: 0 2023-12-06T09:33:47.776 INFO:tasks.ceph.osd.5.smithi132.stderr:2023-12-06T09:33:47.782+0000 ac7d640 -1 osd.5 137 *** Got signal Terminated *** 2023-12-06T09:33:49.313 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:27.292 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.318 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:27.296 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.332 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:27.311 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.364 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:21.679 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.365 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:21.680 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.379 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:21.694 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.387 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:07.733 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.391 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:07.736 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:33:49.406 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:07.752 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.293 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:47.271 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.295 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:47.274 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.309 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:47.288 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.339 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:41.654 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.342 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:27.688 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.342 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:41.657 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.345 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:27.692 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.359 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:27.705 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:09.361 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:07:41.676 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:13.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-06T09:34:18.274 INFO:tasks.ceph.osd.5:Stopped 2023-12-06T09:34:18.275 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-06T09:34:18.275 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:34:18.286 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:34:18.292+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86676) UID: 0 2023-12-06T09:34:18.293 INFO:tasks.ceph.osd.6.smithi132.stderr:2023-12-06T09:34:18.292+0000 ac7d640 -1 osd.6 139 *** Got signal Terminated *** 2023-12-06T09:34:19.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-06T09:34:29.363 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:07.341 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.367 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:07.346 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.385 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:07.364 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.412 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:01.726 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.416 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:47.762 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.416 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:01.731 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.420 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:47.767 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.436 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:01.751 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.437 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:47.783 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.810 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:07.788 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.811 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:07.790 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.824 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:07.803 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.825 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:02.140 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.826 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:02.141 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.832 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:48.178 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.833 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:48.179 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.838 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:02.153 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:29.845 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:48.191 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.388 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:27.366 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.391 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:27.370 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.406 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:27.385 108494== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.435 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:21.750 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.438 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:21.753 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.439 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:07.785 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.442 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:07.788 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.452 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:21.767 108148== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:49.455 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:07.801 124319== Warning: unimplemented fcntl command: 1036 2023-12-06T09:34:59.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-12-06T09:35:00.984 INFO:tasks.ceph.osd.6:Stopped 2023-12-06T09:35:00.985 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-06T09:35:00.985 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:35:00.994 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:35:01.000+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 102297) UID: 0 2023-12-06T09:35:00.994 INFO:tasks.ceph.osd.7.smithi132.stderr:2023-12-06T09:35:01.001+0000 ac7d640 -1 osd.7 139 *** Got signal Terminated *** 2023-12-06T09:35:04.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-06T09:35:04.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-12-06T09:35:07.086 INFO:tasks.ceph.osd.7:Stopped 2023-12-06T09:35:07.087 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-06T09:35:07.087 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-06T09:35:07.087 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:35:07.088 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:35:07.102+0000 7f09f0f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 86663) UID: 0 2023-12-06T09:35:07.088 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-12-06T09:35:07.102+0000 7f09f0f99640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-06T09:35:07.116 INFO:tasks.ceph.mgr.y:Stopped 2023-12-06T09:35:07.116 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-06T09:35:07.116 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:35:07.143 INFO:tasks.ceph.mgr.x:Stopped 2023-12-06T09:35:07.143 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-06T09:35:07.144 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T09:35:07.144 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:35:07.169 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:35:07.174+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 108489) UID: 0 2023-12-06T09:35:07.170 INFO:tasks.ceph.mon.a.smithi088.stderr:2023-12-06T09:35:07.176+0000 9f54640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T09:35:07.696 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:45.674 108494== 2023-12-06T09:35:07.696 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:45.674 108494== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:35:07.710 INFO:tasks.ceph.mon.a.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:35:10.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-06T09:35:10.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-06T09:35:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:35:13.246 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T09:35:13.246 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T09:35:13.247 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T09:35:13.247 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:35:13.269 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:35:13.265+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 124314) UID: 0 2023-12-06T09:35:13.270 INFO:tasks.ceph.mon.c.smithi088.stderr:2023-12-06T09:35:13.274+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T09:35:13.925 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:32.271 124319== 2023-12-06T09:35:13.926 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:32.271 124319== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:35:13.938 INFO:tasks.ceph.mon.c.smithi088.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:35:15.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T09:35:15.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-06T09:35:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:35:19.349 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T09:35:19.349 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T09:35:19.350 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T09:35:19.350 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T09:35:19.366 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:35:19.368+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 108146) UID: 0 2023-12-06T09:35:19.367 INFO:tasks.ceph.mon.b.smithi132.stderr:2023-12-06T09:35:19.370+0000 9f54640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-06T09:35:19.944 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:52.259 108148== 2023-12-06T09:35:19.944 INFO:tasks.ceph.mon.b.smithi132.stderr:==00:00:08:52.259 108148== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T09:35:19.955 INFO:tasks.ceph.mon.b.smithi132.stderr:daemon-helper: command failed with exit status 42 2023-12-06T09:35:20.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T09:35:20.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T09:35:25.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~5s 2023-12-06T09:35:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T09:35:25.451 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T09:35:25.452 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T09:35:25.452 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-06T09:35:25.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW OPS\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-06T09:35:25.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:35:25.551 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-06T09:35:25.655 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:35:25.655 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-06T09:35:25.722 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:35:25.722 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-06T09:35:25.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:35:25.780 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-06T09:35:25.848 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:35:25.848 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-06T09:35:25.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:35:25.962 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-06T09:35:26.019 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:35:26.019 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-06T09:35:26.083 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:35:26.083 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-06T09:35:26.134 INFO:tasks.ceph:Archiving mon data... 2023-12-06T09:35:26.135 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/data/mon.a.tgz 2023-12-06T09:35:26.136 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-12-06T09:35:26.151 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.DPHxf6UZAS 2023-12-06T09:35:26.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DPHxf6UZAS 2023-12-06T09:35:26.286 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.DPHxf6UZAS 2023-12-06T09:35:26.373 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.DPHxf6UZAS is 261KB 2023-12-06T09:35:26.468 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.DPHxf6UZAS 2023-12-06T09:35:26.482 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/data/mon.c.tgz 2023-12-06T09:35:26.483 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-12-06T09:35:26.537 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.uBQsy7l8i4 2023-12-06T09:35:26.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uBQsy7l8i4 2023-12-06T09:35:26.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.uBQsy7l8i4 2023-12-06T09:35:26.734 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.uBQsy7l8i4 is 262KB 2023-12-06T09:35:26.829 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.uBQsy7l8i4 2023-12-06T09:35:26.843 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/data/mon.b.tgz 2023-12-06T09:35:26.844 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2023-12-06T09:35:26.862 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.UmzqoGGONy 2023-12-06T09:35:26.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.UmzqoGGONy 2023-12-06T09:35:27.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.UmzqoGGONy 2023-12-06T09:35:27.087 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.UmzqoGGONy is 262KB 2023-12-06T09:35:27.182 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.UmzqoGGONy 2023-12-06T09:35:27.197 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-06T09:35:27.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-06T09:35:27.200 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-06T09:35:27.526 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T09:35:27.764 INFO:tasks.ceph:Archiving crash dumps... 2023-12-06T09:35:27.766 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/remote/smithi088/crash 2023-12-06T09:35:27.767 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-06T09:35:27.795 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/crash to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/remote/smithi132/crash 2023-12-06T09:35:27.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-06T09:35:27.822 INFO:tasks.ceph:Compressing logs... 2023-12-06T09:35:27.823 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-06T09:35:27.840 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.100381.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86331.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/objectstore_tool.100381.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86331.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.7.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86331.log.gz 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.98687.log 64.0% 2023-12-06T09:35:27.931 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86582.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86491.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/objectstore_tool.98687.log: /var/log/ceph/ceph-client.admin.86582.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86582.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86491.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.86491.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.86291.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86451.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86291.log.gz 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86451.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-06T09:35:27.932 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86451.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86411.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86542.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86411.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86371.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86542.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr: 94.8% -- replaced with /var/log/ceph/objectstore_tool.100381.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.99464.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.86371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86371.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/objectstore_tool.99464.log: 94.7% -- replaced with /var/log/ceph/objectstore_tool.98687.log.gz 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8%gzip -- replaced with /var/log/ceph/ceph.log.gz -5 --verbose -- /var/log/ceph/objectstore_tool.101138.log 2023-12-06T09:35:27.933 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T09:35:27.934 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/objectstore_tool.101138.log: 94.8% -- replaced with /var/log/ceph/objectstore_tool.99464.log.gz 2023-12-06T09:35:27.934 INFO:teuthology.orchestra.run.smithi132.stderr: 94.6% -- replaced with /var/log/ceph/objectstore_tool.101138.log.gz 2023-12-06T09:35:27.934 INFO:teuthology.orchestra.run.smithi132.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-06T09:35:28.055 INFO:teuthology.orchestra.run.smithi132.stderr: 90.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-06T09:35:28.055 INFO:teuthology.orchestra.run.smithi132.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91581.log 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108894.log 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113670.log 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91581.log.gz 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.108894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104893.log 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108894.log.gz 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129686.log 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113670.log.gz 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104893.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.97364.log 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104893.log.gz 2023-12-06T09:35:28.061 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91188.log 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129686.log.gz 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91359.log 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97364.log.gz 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91188.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.91188.log.gz -- 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.92045.log 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104463.log 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91359.log.gz 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105363.log 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92045.log.gz 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94962.log 2023-12-06T09:35:28.062 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104463.log.gz 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.105363.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.98029.log 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105363.log.gz 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94962.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94962.log.gz 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104421.log 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98029.log.gz 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124642.log 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128314.log 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104421.log.gz 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105223.log 2023-12-06T09:35:28.063 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124642.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129182.log 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128314.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.105223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112391.log 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105223.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86970.log 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129182.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127039.log 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112391.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99133.log 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86970.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92129.log 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127039.log.gz 2023-12-06T09:35:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115231.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99133.log.gz 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109431.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92129.log.gz 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120622.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115231.log.gz 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129140.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109431.log.gz 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114502.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120622.log.gz 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127712.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129140.log.gz 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112134.log 2023-12-06T09:35:28.065 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114502.log.gz 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117047.log 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127712.log.gz 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97551.log 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112134.log.gz 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127628.log 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117047.log.gz 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114114.log 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97551.log.gz 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108893.log 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127628.log.gz 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128238.log 2023-12-06T09:35:28.066 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114114.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.108893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105742.log 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr: 29.1% -- replaced with /var/log/ceph/ceph-client.admin.108893.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90775.log 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128238.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.105742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120746.log 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105742.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127926.log 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90775.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120686.log 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120746.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90529.log 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127926.log.gz 2023-12-06T09:35:28.067 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126927.log 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120686.log.gz 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90997.log 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90529.log.gz 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120378.log 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126927.log.gz 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129098.log 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90997.log.gz 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112482.log 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120378.log.gz 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123650.log 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129098.log.gz 2023-12-06T09:35:28.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129014.log 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112482.log.gz 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95620.log 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123650.log.gz 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91239.log 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129014.log.gz 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118826.log 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95620.log.gz 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91239.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91239.log.gz 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.118826.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118826.log.gz 2023-12-06T09:35:28.069 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.2.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123954.log 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122453.log 2023-12-06T09:35:28.070 INFO:teuthology.orchestra.run.smithi088.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92171.log 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123954.log.gz 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116543.log 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122453.log.gz 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128163.log 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92171.log.gz 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128741.log 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116543.log.gz 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127929.log 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128163.log.gz 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124783.log 2023-12-06T09:35:28.071 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128741.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113079.log 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127929.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97912.log 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124783.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114716.log 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113079.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90996.log 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97912.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122691.log 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114716.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129392.log 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90996.log.gz 2023-12-06T09:35:28.072 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127928.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122691.log.gz 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129056.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129392.log.gz 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126269.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127928.log.gz 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99331.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129056.log.gz 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117927.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126269.log.gz 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91280.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99331.log.gz 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127123.log 2023-12-06T09:35:28.073 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117927.log.gz 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91280.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.91280.log.gz 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.127936.log 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90107.log 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127123.log.gz 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104572.log 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127936.log.gz 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114866.log 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90107.log.gz 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105783.log 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104572.log.gz 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122433.log 2023-12-06T09:35:28.074 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114866.log.gz 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.105783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117987.log 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105783.log.gz 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100332.log 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122433.log.gz 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121123.log 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117987.log.gz 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104037.log 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100332.log.gz 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123450.log 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121123.log.gz 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122524.log 2023-12-06T09:35:28.075 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104037.log.gz 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121900.log 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123450.log.gz 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112988.log 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122524.log.gz 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113777.log 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121900.log.gz 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125898.log 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112988.log.gz 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128888.log 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113777.log.gz 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116814.log 2023-12-06T09:35:28.076 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125898.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123390.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128888.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114217.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116814.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94332.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123390.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121091.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114217.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101180.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94332.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123186.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121091.log.gz 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122234.log 2023-12-06T09:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101180.log.gz 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94835.log 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123186.log.gz 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123795.log 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122234.log.gz 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129854.log 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94835.log.gz 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100416.log 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123795.log.gz 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90999.log 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129854.log.gz 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91417.log 2023-12-06T09:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100416.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109949.log 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90999.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125427.log 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91417.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125765.log 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109949.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91711.log 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125427.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94856.log 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125765.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111946.log 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91711.log.gz 2023-12-06T09:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91802.log 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94856.log.gz 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.111946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116745.log 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111946.log.gz 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98672.log 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91802.log.gz 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121849.log 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116745.log.gz 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129350.log 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98672.log.gz 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124966.log 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121849.log.gz 2023-12-06T09:35:28.080 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104222.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129350.log.gz 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123734.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124966.log.gz 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98893.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104222.log.gz 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127924.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123734.log.gz 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110085.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98893.log.gz 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129518.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127924.log.gz 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.110085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126928.log 2023-12-06T09:35:28.081 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110085.log.gz 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128588.log 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129518.log.gz 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92761.log 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126928.log.gz 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91631.log 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128588.log.gz 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102119.log 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92761.log.gz 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111563.log 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91631.log.gz 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.102119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113568.log 2023-12-06T09:35:28.082 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102119.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.111563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99013.log 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111563.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91000.log 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113568.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126864.log 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99013.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112896.log 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91000.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91919.log 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126864.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126399.log 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112896.log.gz 2023-12-06T09:35:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92846.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91919.log.gz 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112327.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126399.log.gz 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94756.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92846.log.gz 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112327.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112327.log.gz 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129812.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94756.log.gz 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115292.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122861.log 2023-12-06T09:35:28.084 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129812.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124907.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115292.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110523.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122861.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119892.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124907.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.110523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88852.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110523.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.119892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98490.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119892.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90635.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88852.log.gz 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117586.log 2023-12-06T09:35:28.085 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98490.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128508.log 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90635.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117586.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128174.log 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114928.log 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128508.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128508.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128174.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108594.log 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116404.log 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114928.log.gz 2023-12-06T09:35:28.086 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.108594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126824.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108594.log.gz 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112926.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116404.log.gz 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92639.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126824.log.gz 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124824.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112926.log.gz 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108493.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92639.log.gz 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114013.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124824.log.gz 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.108493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129728.log 2023-12-06T09:35:28.087 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108493.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95732.log 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114013.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115576.log 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129728.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117308.log/var/log/ceph/ceph-client.admin.95732.log: 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95732.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91541.log 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115576.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115431.log 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117308.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124198.log 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91541.log.gz 2023-12-06T09:35:28.088 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125897.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115431.log.gz 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120854.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124198.log.gz 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92510.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125897.log.gz 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122580.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120854.log.gz 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97985.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92510.log.gz 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121450.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122580.log.gz 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112772.log 2023-12-06T09:35:28.089 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97985.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121235.log 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121450.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120410.log 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112772.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122841.log 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121235.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114786.log 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120410.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121706.log 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122841.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127081.log 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114786.log.gz 2023-12-06T09:35:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129770.log 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121706.log.gz 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101077.log 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127081.log.gz 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100742.log 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129770.log.gz 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101077.log.gz 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116364.log 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128678.log 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100742.log.gz 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.116364.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.93229.log 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116364.log.gz 2023-12-06T09:35:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.128678.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95078.log 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128678.log.gz 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.93229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96076.log 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93229.log.gz 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125971.log 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95078.log.gz 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98349.log 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96076.log.gz 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119851.log 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125971.log.gz 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105619.log 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98349.log.gz 2023-12-06T09:35:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.119851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123045.log 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119851.log.gz 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.105619.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105619.log.gz 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95226.log 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123045.log.gz 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87857.log 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95226.log.gz 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111771.log 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127586.log 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87857.log.gz 2023-12-06T09:35:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.111771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92087.log 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111771.log.gz 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119561.log 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127586.log.gz 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92087.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92087.log.gz -5 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.129602.log 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.119561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125166.log 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119561.log.gz 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109704.log 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129602.log.gz 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128844.log 2023-12-06T09:35:28.094 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125166.log.gz 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109704.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.109704.log.gz /var/log/ceph/ceph-client.admin.93852.log 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127333.log 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128844.log.gz 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.93852.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.93852.log.gz -- 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.94575.log 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129476.log 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127333.log.gz 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127460.log 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94575.log.gz 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90687.log 2023-12-06T09:35:28.095 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129476.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124096.log 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127460.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96501.log 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90687.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111820.log 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124096.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128311.log 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96501.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.111820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111820.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91358.log 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128311.log.gz 2023-12-06T09:35:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97446.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121989.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91358.log.gz 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91691.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97446.log.gz 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111025.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121989.log.gz 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113202.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91691.log.gz 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.111025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122010.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111025.log.gz 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113202.log.gz 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116792.log 2023-12-06T09:35:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100262.log 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122010.log.gz 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116792.log.gz 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116063.log 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105182.log 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100262.log.gz 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116063.log.gz 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113122.log 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.105182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104507.log 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105182.log.gz 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113122.log.gz 2023-12-06T09:35:28.098 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117267.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90569.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104507.log.gz 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117267.log.gz 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127670.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120788.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90569.log.gz 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91601.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127670.log.gz 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118187.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120788.log.gz 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128546.log 2023-12-06T09:35:28.099 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91601.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.118187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98543.log 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118187.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128546.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127375.log 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96541.log 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98543.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106078.log 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127375.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129644.log 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96541.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.106078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128648.log 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106078.log.gz 2023-12-06T09:35:28.100 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101318.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129644.log.gz 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128648.log.gz 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87173.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101318.log.gz 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114269.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87173.log.gz 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124240.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116175.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114269.log.gz 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112046.log 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124240.log.gz 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116175.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.116175.log.gz --verbose 2023-12-06T09:35:28.101 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.100960.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127165.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112046.log.gz 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100960.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.100960.log.gz -5 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101602.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91877.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127165.log.gz 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125487.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101602.log.gz 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91877.log.gz 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96823.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112611.log 2023-12-06T09:35:28.102 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125487.log.gz 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110207.log 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96823.log.gz 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112611.log.gz 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126356.log 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92357.log 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.110207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110207.log.gz 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126356.log.gz 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129560.log 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99608.log 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92357.log.gz 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104202.log 2023-12-06T09:35:28.103 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129560.log.gz 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99608.log.gz 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96863.log 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.104202.log: /var/log/ceph/ceph-client.admin.116240.log 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104202.log.gz 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96863.log.gz 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109091.log 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118007.log 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116240.log.gz 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95903.log 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109091.log.gz 2023-12-06T09:35:28.104 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.118007.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.118007.log.gz -- 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.127923.log 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120002.log 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95903.log.gz 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122235.log 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127923.log.gz 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127544.log 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120002.log.gz 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104848.log 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122235.log.gz 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118746.log 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127544.log.gz 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.104848.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.104848.log.gz 2023-12-06T09:35:28.105 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.92233.log 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126620.log 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.118746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118746.log.gz 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115778.log 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92233.log.gz 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126620.log.gz 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123671.log 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127935.log 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115778.log.gz 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123671.log.gz 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115798.log 2023-12-06T09:35:28.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128242.log 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127935.log.gz 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101819.log 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115798.log.gz 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128242.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.128242.log.gz -- 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.126139.log 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127754.log 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101819.log.gz 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126139.log.gz 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95284.log 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114320.log 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127754.log.gz 2023-12-06T09:35:28.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95284.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.95284.log.gz 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.99870.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114320.log.gz 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114601.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95780.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99870.log.gz 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114601.log.gz 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128628.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115333.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95780.log.gz 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124582.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128628.log.gz 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123225.log 2023-12-06T09:35:28.108 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115333.log.gz 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123085.log 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124582.log.gz 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123225.log.gz 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92979.log 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123085.log.gz 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115898.log 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121521.log 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92979.log.gz 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93610.log 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115898.log.gz 2023-12-06T09:35:28.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92356.log 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121521.log.gz 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.93610.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93610.log.gz 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101459.log 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92356.log.gz 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92353.log 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101459.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101459.log.gz 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90995.log 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92353.log.gz 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113952.log 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90995.log.gz 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120665.log 2023-12-06T09:35:28.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125663.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113952.log.gz 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120665.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120665.log.gz 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.129434.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125663.log.gz 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109308.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128804.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129434.log.gz 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121553.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109308.log.gz 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91961.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128804.log.gz 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126442.log 2023-12-06T09:35:28.111 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121553.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128972.log 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91961.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120299.log 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126442.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87073.log 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128972.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114828.log 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120299.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120875.log 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87073.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100899.log 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114828.log.gz 2023-12-06T09:35:28.112 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112221.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120875.log.gz 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120069.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100899.log.gz 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125337.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112221.log.gz 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127930.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120069.log.gz 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123585.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125337.log.gz 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129266.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127930.log.gz 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109579.log 2023-12-06T09:35:28.113 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123585.log.gz 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97770.log 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129266.log.gz 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92907.log 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109579.log.gz 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109197.log 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97770.log.gz 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99670.log 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92907.log.gz 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128188.log 2023-12-06T09:35:28.114 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109197.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97593.log 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99670.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94714.log 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128188.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127291.log 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97593.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113037.log 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94714.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121379.log 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127291.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93127.log 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113037.log.gz 2023-12-06T09:35:28.115 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113203.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121379.log.gz 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91001.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.93127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93127.log.gz 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113203.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113203.log.gz 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114482.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91001.log.gz 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117225.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114482.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114482.log.gz 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.109785.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117225.log.gz 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117688.log 2023-12-06T09:35:28.116 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.109785.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.115941.log 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109785.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99991.log 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117688.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.115941.log: --verbose -- /var/log/ceph/ceph-client.admin.107112.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115941.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.99991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99991.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111684.log 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.107112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107112.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115718.log 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.111684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111684.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127502.log 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115718.log.gz 2023-12-06T09:35:28.117 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120136.log 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127502.log.gz 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125381.log 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120136.log.gz 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126680.log 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125381.log.gz 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115063.log 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.126680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126680.log.gz 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118424.log 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115063.log.gz 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128468.log 2023-12-06T09:35:28.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113630.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.118424.log: gzip/var/log/ceph/ceph-client.admin.128468.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.96009.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118424.log.gz 0.0% 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128468.log.gz 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113630.log.gz 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116885.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.96009.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96009.log.gz 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.93331.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.116885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116885.log.gz 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80845.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.93331.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.113389.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93331.log.gz 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.tmp-client.admin.80845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96204.log 2023-12-06T09:35:28.119 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80845.log.gz 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113389.log: /var/log/ceph/ceph-client.admin.96204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124318.log 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113389.log.gz 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96204.log.gz/var/log/ceph/ceph.log: 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98223.log 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124318.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92003.log 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95660.log 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98223.log.gz 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92003.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92003.log.gz 2023-12-06T09:35:28.120 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122073.log 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95660.log.gz 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120534.log 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122073.log.gz 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129936.log 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr: 93.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.120534.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86666.log 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.120534.log.gz 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129936.log.gz 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121451.log 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104957.log 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86666.log.gz 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121451.log.gzgzip 2023-12-06T09:35:28.121 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.124946.log 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91803.log 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124946.log: /var/log/ceph/ceph-client.admin.104957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104957.log.gz 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124946.log.gz 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98951.log 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121319.log 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91803.log.gz 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.98951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98951.log.gz 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127249.log 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100677.log 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121319.log: /var/log/ceph/ceph-client.admin.127249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127249.log.gz 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121319.log.gz 2023-12-06T09:35:28.122 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93530.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100677.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113712.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129896.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113712.log: /var/log/ceph/ceph-client.admin.93530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93530.log.gz 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113712.log.gz 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127417.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92804.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129896.log.gz 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.127417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127417.log.gz 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122620.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101222.log 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.122620.log: /var/log/ceph/ceph-client.admin.92804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92804.log.gz 2023-12-06T09:35:28.123 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122620.log.gz 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92908.log 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97239.log 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.101222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101222.log.gz 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.92908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92908.log.gz 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129224.log 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94515.log 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.97239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97239.log.gz 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129224.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123137.log 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.129224.log.gz 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.94515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93352.log 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94515.log.gz 2023-12-06T09:35:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.123137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129308.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0%/var/log/ceph/ceph-client.admin.93352.log: -- replaced with /var/log/ceph/ceph-client.admin.123137.log.gz 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93352.log.gz 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119688.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126992.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.129308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129308.log.gz 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.119688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119688.log.gz 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105465.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128930.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.105465.log: /var/log/ceph/ceph-client.admin.126992.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126992.log.gz 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105465.log.gz 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119933.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110257.log 2023-12-06T09:35:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128930.log.gz 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.119933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119933.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90819.log 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124460.log 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.110257.log: /var/log/ceph/ceph-client.admin.90819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90819.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110257.log.gz 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124116.log 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120137.log 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124460.log.gz 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124116.log.gz 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90998.log 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113368.log 2023-12-06T09:35:28.126 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90998.log: /var/log/ceph/ceph-client.admin.120137.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120137.log.gz 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90998.log.gz 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110703.log 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112431.log 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.113368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113368.log.gz 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.110703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110703.log.gz 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95327.log 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91233.log 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.95327.log: /var/log/ceph/ceph-client.admin.112431.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112431.log.gz 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95327.log.gz 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118524.log 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98138.log 2023-12-06T09:35:28.127 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91233.log.gz 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.118524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118524.log.gz 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108752.log 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124741.log 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.108752.log: /var/log/ceph/ceph-client.admin.98138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108752.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98138.log.gz 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112662.log 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127207.log 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.124741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124741.log.gz 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.112662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112662.log.gz 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115373.log 2023-12-06T09:35:28.128 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90993.log 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.115373.log: /var/log/ceph/ceph-client.admin.127207.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115373.log.gz 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.127207.log.gz 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91273.log 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128153.log 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90993.log.gz 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.91273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91273.log.gz 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121299.log 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.128153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128153.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125294.log 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.121299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114157.log 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.125294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125294.log.gz 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121299.log.gz 2023-12-06T09:35:28.129 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90733.log 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.114157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114157.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102759.log 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.90733.log: /var/log/ceph/ceph-client.admin.102759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102759.log.gz 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90733.log.gz 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100597.log 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117646.log 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.100597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100597.log.gz 2023-12-06T09:35:28.130 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.117646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117646.log.gz 2023-12-06T09:35:28.131 INFO:teuthology.orchestra.run.smithi132.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-06T09:35:28.131 INFO:teuthology.orchestra.run.smithi132.stderr: 90.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-06T09:35:28.131 INFO:teuthology.orchestra.run.smithi088.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-06T09:35:28.132 INFO:teuthology.orchestra.run.smithi088.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-06T09:35:28.132 INFO:teuthology.orchestra.run.smithi088.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-06T09:35:28.143 INFO:teuthology.orchestra.run.smithi088.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-06T09:35:28.150 INFO:teuthology.orchestra.run.smithi088.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-06T09:35:28.162 INFO:teuthology.orchestra.run.smithi132.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-06T09:35:28.164 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T09:35:28.164 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m0.282s 2023-12-06T09:35:28.164 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m1.002s 2023-12-06T09:35:28.164 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m0.061s 2023-12-06T09:35:28.251 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-06T09:35:28.306 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-06T09:35:28.308 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-12-06T09:35:28.308 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.457s 2023-12-06T09:35:28.308 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m1.663s 2023-12-06T09:35:28.308 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.236s 2023-12-06T09:35:28.309 INFO:tasks.ceph:Archiving logs... 2023-12-06T09:35:28.309 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/remote/smithi088/log 2023-12-06T09:35:28.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-06T09:35:28.926 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/remote/smithi132/log 2023-12-06T09:35:28.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-06T09:35:29.131 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T09:35:29.132 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-06T09:35:29.141 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T09:35:29.142 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-06T09:35:29.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-06T09:35:29.145 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-06T09:35:29.224 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-06T09:35:29.233 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-06T09:35:29.234 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T09:35:29.236 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T09:35:29.249 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2023-12-06T09:35:29.251 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2023-12-06T09:35:29.253 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T09:35:29.253 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-12-06T09:35:29.253 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1464us[+1464us] +/- 96ms 2023-12-06T09:35:29.253 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 7 377 118 -797us[ -804us] +/- 59ms 2023-12-06T09:35:29.253 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -3839ns[-3839ns] +/- 85ms 2023-12-06T09:35:29.253 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:35:29.255 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T09:35:29.255 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2023-12-06T09:35:29.255 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1435us[+1435us] +/- 96ms 2023-12-06T09:35:29.255 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -839us[ -832us] +/- 60ms 2023-12-06T09:35:29.255 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -31us[ -25us] +/- 85ms 2023-12-06T09:35:29.255 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T09:35:29.256 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-06T09:35:29.265 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-06T09:35:29.265 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-06T09:35:29.274 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-06T09:35:29.347 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-06T09:35:29.459 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T09:35:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:35:29.488 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T09:35:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:35:29.521 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-06T09:35:29.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-06T09:35:29.540 INFO:teuthology.task.internal:Duration was 1826.660085 seconds 2023-12-06T09:35:29.540 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-06T09:35:29.549 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-06T09:35:29.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-06T09:35:29.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-06T09:35:29.593 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T09:35:29.605 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T09:35:29.869 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-06T09:35:29.870 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-12-06T09:35:29.870 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-06T09:35:29.889 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T09:35:29.889 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-06T09:35:29.909 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-06T09:35:29.909 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-06T09:35:29.931 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-06T09:35:29.977 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-06T09:35:29.978 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-06T09:35:29.999 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-06T09:35:30.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-06T09:35:30.154 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-06T09:35:30.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-06T09:35:30.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T09:35:30.181 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-06T09:35:30.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-06T09:35:30.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-06T09:35:30.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-06T09:35:30.247 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-12-06T09:35:30.274 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2023-12-06T09:35:30.296 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-06T09:35:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:35:30.331 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-06T09:35:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T09:35:30.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-06T09:35:30.366 INFO:teuthology.task.internal:Transferring archived files... 2023-12-06T09:35:30.367 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/remote/smithi088 2023-12-06T09:35:30.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-06T09:35:30.404 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622/remote/smithi132 2023-12-06T09:35:30.404 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-06T09:35:30.443 INFO:teuthology.task.internal:Removing archive directory... 2023-12-06T09:35:30.444 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-06T09:35:30.446 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-06T09:35:30.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-06T09:35:30.504 INFO:teuthology.task.internal:Not uploading archives. 2023-12-06T09:35:30.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-06T09:35:30.603 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-06T09:35:30.603 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-06T09:35:30.606 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-06T09:35:30.618 INFO:teuthology.orchestra.run.smithi088.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 6 09:35 /home/ubuntu/cephtest 2023-12-06T09:35:30.620 INFO:teuthology.orchestra.run.smithi132.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 6 09:35 /home/ubuntu/cephtest 2023-12-06T09:35:30.621 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-06T09:35:30.631 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-06T09:35:30.642 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T09:35:30.642 INFO:teuthology.nuke:Checking targets against current locks 2023-12-06T09:35:30.668 DEBUG:teuthology.nuke:shortname: smithi088 2023-12-06T09:35:30.668 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T09:35:30.691 DEBUG:teuthology.nuke:shortname: smithi132 2023-12-06T09:35:30.691 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T09:35:30.695 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 08:55:42.840638', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHHTCpfjfHqstmkXRAr29MBgjiE9LKJ7mmVdbETF57yMeJIhqBvVHbaKtJ/N+dGuo6+PDjx/O8tQ0aC31Q0dv0='} 2023-12-06T09:35:30.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 08:55:42.842083', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwTJzvIaMcfYCeVIdChPPodcm3xbZK0f0XHf0RcDpgGKOBNDbbi3jNe6hMMeG4nf8PxhSd4BZBMjCWL9Zi/qqA='} 2023-12-06T09:35:30.746 INFO:teuthology.orchestra.console.smithi088:Power off 2023-12-06T09:35:30.746 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-06T09:35:30.761 INFO:teuthology.orchestra.console.smithi132:Power off 2023-12-06T09:35:30.761 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-06T09:35:30.770 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2023-12-06T09:35:30.770 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T09:35:30.784 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2023-12-06T09:35:30.784 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T09:35:30.794 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-12-06T09:35:30.814 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2023-12-06T09:35:34.796 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T09:35:34.815 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T09:35:34.906 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-12-06T09:35:34.926 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2023-12-06T09:35:38.910 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T09:35:38.927 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T09:35:39.020 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2023-12-06T09:35:39.037 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2023-12-06T09:35:39.121 INFO:teuthology.orchestra.console.smithi088:Power off completed 2023-12-06T09:35:39.138 INFO:teuthology.orchestra.console.smithi132:Power off completed 2023-12-06T09:35:39.242 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} duration: 1826.6600849628448 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate' flavor: default owner: scheduled_sseshasa@teuthology status: fail success: false 2023-12-06T09:35:39.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T09:35:39.335 INFO:teuthology.run:FAIL